www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Win32 headers: Problem with COM interface indirection levels

reply Michael Butscher <mbutscher gmx.de> writes:
Hi,

there seems to be a rather fundamental error in the translation of COM 
interfaces by the Win32 bindings project:

The C++ headers defines an interface as a class (more precisely a 
struct) with some virtual methods. This means, an instance of the 
interface is the actual object data, mainly containing a pointer to the 
vtable.

In D, an interface is a pointer to the object data which contains in 
turn the pointer to the vtable.

Because of the different level of indirection e.g. the IShellFolder 
interface in D is in fact equivalent to the LPSHELLFOLDER type in C++.


I think there are two possible solutions:

1. Modify parameter types of all functions using the interface, e.g. 
translate C++ definition:

  HRESULT SHGetDesktopFolder(LPSHELLFOLDER*);


to:
  
  HRESULT SHGetDesktopFolder(LPSHELLFOLDER);

or:

  HRESULT SHGetDesktopFolder(IShellFolder*);




2. Rewrite the definitions of the LP... types as aliases of the 
interfaces (I would prefer that), e.g.

C++:

  typedef IShellFolder *LPSHELLFOLDER;


to D:

  alias IShellFolder LPSHELLFOLDER;




I think this is important so the decision should be documented in the 
Translation instructions.



Michael
Aug 08 2007
next sibling parent reply Carlos Santander <csantander619 gmail.com> writes:
Michael Butscher escribió:
 Hi,
 
 there seems to be a rather fundamental error in the translation of COM 
 interfaces by the Win32 bindings project:
 
 The C++ headers defines an interface as a class (more precisely a 
 struct) with some virtual methods. This means, an instance of the 
 interface is the actual object data, mainly containing a pointer to the 
 vtable.
 
 In D, an interface is a pointer to the object data which contains in 
 turn the pointer to the vtable.
 
 Because of the different level of indirection e.g. the IShellFolder 
 interface in D is in fact equivalent to the LPSHELLFOLDER type in C++.
 
 
 I think there are two possible solutions:
 
 1. Modify parameter types of all functions using the interface, e.g. 
 translate C++ definition:
 
   HRESULT SHGetDesktopFolder(LPSHELLFOLDER*);
 
 
 to:
   
   HRESULT SHGetDesktopFolder(LPSHELLFOLDER);
 
 or:
 
   HRESULT SHGetDesktopFolder(IShellFolder*);
 
 
 
 
 2. Rewrite the definitions of the LP... types as aliases of the 
 interfaces (I would prefer that), e.g.
 
 C++:
 
   typedef IShellFolder *LPSHELLFOLDER;
 
 
 to D:
 
   alias IShellFolder LPSHELLFOLDER;
 
 
 
 
 I think this is important so the decision should be documented in the 
 Translation instructions.
 
 
 
 Michael

I don't use Windows, but I haven't heard of anyone having this problem before. Can you test that with a simpler case? Remember that there's a ComInterface (or something like that) in std.c.windows.com (I think) which gets special treatment by the compiler. Maybe the compiler performs some magic there in order to get everything working fine. Also, maybe this question is better suited in the Bindings project Dsource forum? -- Carlos Santander Bernal
Aug 08 2007
parent reply Michael Butscher <mbutscher gmx.de> writes:
Carlos Santander wrote:
 Michael Butscher escribió:

 Because of the different level of indirection e.g. the IShellFolder 
 interface in D is in fact equivalent to the LPSHELLFOLDER type in C++.


 I don't use Windows, but I haven't heard of anyone having this problem before.

This might be an indication that this part of the headers is currently not used widely (maybe by some people changing their copy of the headers privately). But the published version definitely can't work.
 Can you test that with a simpler case?

I have made a little test already. The provided headers lead to a access violation. Modifying it according to my request make them work. I found out in the meantime that some headers (namely "d3d9.d") were modified already in a similar way, but it was not documented in the translation instructions.
 Remember that there's a ComInterface (or 
 something like that) in std.c.windows.com (I think) which gets special
treatment 
 by the compiler.

It seems that every interface named "IUnknown" (and derived interfaces) is handled specially, regardless of the containing module. This is not the problem.
 Maybe the compiler performs some magic there in order to get 
 everything working fine.
 
 Also, maybe this question is better suited in the Bindings project Dsource
forum?

You are right. I only found forums a bit impractical, but that's not a good argument. Michael
Aug 09 2007
next sibling parent reply jcc7 <technocrat7 gmail.com> writes:
== Quote from Michael Butscher (mbutscher gmx.de)'s article
 Carlos Santander wrote:
 Michael Butscher escribió:


 I found out in the meantime that some headers (namely
 "d3d9.d") were modified already in a similar way, but it was not
 documented in the translation instructions.

It's been mentioned on the wiki page at http://www.prowiki.org/wiki4d/wiki.cgi?WindowsAPI, but I think you're right that it's not in the "Translation instructions" section (and it probably should be there). I get the feeling that the WindowsAPI page at Wiki4D has gotten way too big to be efficient. I think that it should probably be split into subpages, but I would be afraid of stepping on the toes of the WindowsAPI team by doing it myself. (Also, I wonder if it would work better to store the content that's current at Wiki4D in the bindings wiki instead [http://www.dsource.org/projects/bindings], since it is part of the bindings project and dsource wikis have a cool way of formatting D source code.)
 Remember that there's a ComInterface (or
 something like that) in std.c.windows.com (I think) which gets
 special treatment by the compiler.

interfaces) is handled specially, regardless of the containing module. This is not the problem.
 Maybe the compiler performs some magic there in order to get
 everything working fine.

 Also, maybe this question is better suited in the Bindings project
 Dsource forum?

a good argument.

Actually by my count, it's already been mentioned twice by other people in the Bindings forum. http://www.dsource.org/forums/viewtopic.php?t=2458 http://www.dsource.org/forums/viewtopic.php?t=2253#14074 (The bindings project doesn't really have formal leadership, though, so I'm not sure who's reading that forum anyway.)
Aug 09 2007
next sibling parent reply Michael Butscher <mbutscher gmx.de> writes:
jcc7 wrote:
 == Quote from Michael Butscher (mbutscher gmx.de)'s article
 Carlos Santander wrote:
 Michael Butscher escribió:


 I found out in the meantime that some headers (namely
 "d3d9.d") were modified already in a similar way, but it was not
 documented in the translation instructions.

It's been mentioned on the wiki page at http://www.prowiki.org/wiki4d/wiki.cgi?WindowsAPI, but I think you're right that it's not in the "Translation instructions" section (and it probably should be there). I get the feeling that the WindowsAPI page at Wiki4D has gotten way too big to be efficient. I think that it should probably be split into subpages, but I would be afraid of stepping on the toes of the WindowsAPI team by doing it myself.

Personally I prefer one large page as I'm using a dialup connection, so I have to load only one page and can read it offline easily.
 (Also, I wonder if it would work better to store the content that's current at
 Wiki4D in the bindings wiki instead [http://www.dsource.org/projects/bindings],
 since it is part of the bindings project and dsource wikis have a cool way of
 formatting D source code.)

At least there should be only one place for the instructions. Currently there exists also a readme file in the SVN-repository with similar instructions. Either the file should point to the wiki page(s) or vice versa. [...]
 Also, maybe this question is better suited in the Bindings project
 Dsource forum?

a good argument.

Actually by my count, it's already been mentioned twice by other people in the Bindings forum. http://www.dsource.org/forums/viewtopic.php?t=2458 http://www.dsource.org/forums/viewtopic.php?t=2253#14074

I thought that somebody would have written it in the instructions if the problem was realized already therefore I didn't search for it in the forum. Michael
Aug 10 2007
next sibling parent reply jcc7 <technocrat7 gmail.com> writes:
== Quote from Michael Butscher (mbutscher gmx.de)'s article
 jcc7 wrote:
 == Quote from Michael Butscher (mbutscher gmx.de)'s article
 Carlos Santander wrote:
 Michael Butscher escribió:


 I found out in the meantime that some headers (namely
 "d3d9.d") were modified already in a similar way, but it was not
 documented in the translation instructions.

It's been mentioned on the wiki page at http://www.prowiki.org/wiki4d/wiki.cgi?WindowsAPI, but I think you're right that it's not in the "Translation instructions" section (and it probably should be


 I get the feeling that the WindowsAPI page at Wiki4D has gotten way too big to
be
 efficient. I think that it should probably be split into subpages, but I would
be
 afraid of stepping on the toes of the WindowsAPI team by doing it myself.

I have to load only one page and can read it offline easily.

I understand the reason for your preference, but I still think the page is way too long. Perhaps the instructions could stay on the main page, but the list of contributor and the discussions could be kept on separate pages. Anyway, I don't want to interfere with the project, so I'll let someone actually involved in the project make that change if they want to take my (unsolicited) advice.
 (Also, I wonder if it would work better to store the content that's current at
 Wiki4D in the bindings wiki instead [http://www.dsource.org/projects/bindings],
 since it is part of the bindings project and dsource wikis have a cool way of
 formatting D source code.)

there exists also a readme file in the SVN-repository with similar instructions.

That does sound like a problem.
 Either the file should point to the wiki page(s) or vice versa.

I agree. Personally, I would prefer to have a wiki page as the sole place due to the greater options for formatting.
 [...]
 Also, maybe this question is better suited in the Bindings project
 Dsource forum?

a good argument.

Actually by my count, it's already been mentioned twice by other people in the Bindings forum. http://www.dsource.org/forums/viewtopic.php?t=2458 http://www.dsource.org/forums/viewtopic.php?t=2253#14074

the problem was realized already therefore I didn't search for it in the forum. Michael

I don't know how the people API project are talking to each other. I assumed it was via the wiki page, but for all I know they could be using IRC, AIM, or an e-mail mailing list. I was just trying to make the point that apparently it doesn't matter if anyone posts in the bindings forum because the people who might want to know that information don't seem to even be reading that forum. I'm not a leader in the API project (or even a member). I expect that the leadership probably doesn't want just anyone editing their instructions section. I suspect that why the people that posted on forum didn't edit the instructions themselves. Or maybe they weren't aware of the Wiki4D page? (I'm not sure how the API project is organized.) I did at least make a comment on that page, but it's become such an unwieldy page that I don't think anyone has read it from beginning to end recently.
Aug 10 2007
next sibling parent reply Michael Butscher <mbutscher gmx.de> writes:
jcc7 wrote:
 == Quote from Michael Butscher (mbutscher gmx.de)'s article
 jcc7 wrote:


 I get the feeling that the WindowsAPI page at Wiki4D has gotten way too big to
be
 efficient. I think that it should probably be split into subpages, but I would
be
 afraid of stepping on the toes of the WindowsAPI team by doing it myself.

I have to load only one page and can read it offline easily.

I understand the reason for your preference, but I still think the page is way too long. Perhaps the instructions could stay on the main page, but the list of contributor and the discussions could be kept on separate pages. Anyway, I don't want to interfere with the project, so I'll let someone actually involved in the project make that change if they want to take my (unsolicited) advice.

Now I have posted a message to the Bindings forum about that. Let's see what happens. [...]
 I thought that somebody would have written it in the instructions if
 the problem was realized already therefore I didn't search for it in
 the forum.
 Michael

I don't know how the people API project are talking to each other. I assumed it was via the wiki page, but for all I know they could be using IRC, AIM, or an e-mail mailing list. I was just trying to make the point that apparently it doesn't matter if anyone posts in the bindings forum because the people who might want to know that information don't seem to even be reading that forum. I'm not a leader in the API project (or even a member). I expect that the leadership probably doesn't want just anyone editing their instructions section. I suspect that why the people that posted on forum didn't edit the instructions themselves. Or maybe they weren't aware of the Wiki4D page? (I'm not sure how the API project is organized.)

The start page of the bindings project wiki has a link to the Wiki4D page, so the "WindowsAPI" page seems to be "official". Michael
Aug 11 2007
parent reply Justin C Calvarese <technocrat7 gmail.com> writes:
Michael Butscher wrote:
 jcc7 wrote:
 == Quote from Michael Butscher (mbutscher gmx.de)'s article
 jcc7 wrote:


 I get the feeling that the WindowsAPI page at Wiki4D has gotten way too big to
be
 efficient. I think that it should probably be split into subpages, but I would
be
 afraid of stepping on the toes of the WindowsAPI team by doing it myself.

I have to load only one page and can read it offline easily.

long. Perhaps the instructions could stay on the main page, but the list of contributor and the discussions could be kept on separate pages. Anyway, I don't want to interfere with the project, so I'll let someone actually involved in the project make that change if they want to take my (unsolicited) advice.

Now I have posted a message to the Bindings forum about that. Let's see what happens.

Fair enough.
 [...]
 I thought that somebody would have written it in the instructions if
 the problem was realized already therefore I didn't search for it in
 the forum.
 Michael

was via the wiki page, but for all I know they could be using IRC, AIM, or an e-mail mailing list. I was just trying to make the point that apparently it doesn't matter if anyone posts in the bindings forum because the people who might want to know that information don't seem to even be reading that forum. I'm not a leader in the API project (or even a member). I expect that the leadership probably doesn't want just anyone editing their instructions section. I suspect that why the people that posted on forum didn't edit the instructions themselves. Or maybe they weren't aware of the Wiki4D page? (I'm not sure how the API project is organized.)

The start page of the bindings project wiki has a link to the Wiki4D page, so the "WindowsAPI" page seems to be "official".

Actually, I was the one who added that link, but it's been over a year, so if anyone objected, I hope they would've corrected it by now. ;) http://www.dsource.org/projects/bindings/wiki/WikiStart?action=diff&version=6&old_version=5 -- jcc7
Aug 11 2007
parent "Stewart Gordon" <smjg_1998 yahoo.com> writes:
"Justin C Calvarese" <technocrat7 gmail.com> wrote in message 
news:f9lsdf$m5b$1 digitalmars.com...
 Michael Butscher wrote:

 The start page of the bindings project wiki has a link to the Wiki4D 
 page, so the "WindowsAPI" page seems to be "official".

Actually, I was the one who added that link, but it's been over a year, so if anyone objected, I hope they would've corrected it by now. ;) http://www.dsource.org/projects/bindings/wiki/WikiStart?action=diff&version=6&old_version=5

The page on Wiki4D is indeed the official one at the moment. Stewart.
Aug 27 2007
prev sibling parent "Stewart Gordon" <smjg_1998 yahoo.com> writes:
"jcc7" <technocrat7 gmail.com> wrote in message 
news:f9hu7a$2j79$1 digitalmars.com...
<snip>
 I don't know how the people API project are talking to each other. I 
 assumed it
 was via the wiki page, but for all I know they could be using IRC, AIM, or 
 an
 e-mail mailing list. I was just trying to make the point that apparently 
 it
 doesn't matter if anyone posts in the bindings forum because the people 
 who might
 want to know that information don't seem to even be reading that forum.

It is mostly via the wiki page and these newsgroups. The problem is that not enough people are working on it, and those that do tend to disappear after a while. Stewart.
Aug 27 2007
prev sibling parent "Stewart Gordon" <smjg_1998 yahoo.com> writes:
"Michael Butscher" <mbutscher gmx.de> wrote in message 
news:MPG.21268ee9722f11e99896a9 news.digitalmars.com...
<snip>
 At least there should be only one place for the instructions. Currently
 there exists also a readme file in the SVN-repository with similar
 instructions.

They're meant to be the same, but probably have got a bit out of sync.
 Either the file should point to the wiki page(s) or vice versa.

Having it in the SVN at least means that we have access to a revision history of the instructions. But migrating it onto the Dsource wiki would be one way to avoid losing this. Good thinking, whosever idea this was! Stewart.
Aug 27 2007
prev sibling parent reply "Stewart Gordon" <smjg_1998 yahoo.com> writes:
"jcc7" <technocrat7 gmail.com> wrote in message 
news:f9f7oc$2ovj$1 digitalmars.com...
<snip>
 It's been mentioned on the wiki page at
 http://www.prowiki.org/wiki4d/wiki.cgi?WindowsAPI, but I think you're 
 right that
 it's not in the "Translation instructions" section (and it probably should 
 be there).

 I get the feeling that the WindowsAPI page at Wiki4D has gotten way too 
 big to be
 efficient. I think that it should probably be split into subpages, but I 
 would be
 afraid of stepping on the toes of the WindowsAPI team by doing it myself.

It probably could be split, though I'm not sure what would be the best approach.
 (Also, I wonder if it would work better to store the content that's 
 current at
 Wiki4D in the bindings wiki instead 
 [http://www.dsource.org/projects/bindings],
 since it is part of the bindings project and dsource wikis have a cool way 
 of
 formatting D source code.)

Maybe that would be a good idea. I haven't used Dsource's wiki engine much - how does it compare to prowiki? At least it seems able to show arbitrary versions of a page like MediaWiki, rather than just the current version and the most recent edit. Stewart.
Aug 27 2007
parent jcc7 <technocrat7 gmail.com> writes:
== Quote from Stewart Gordon (smjg_1998 yahoo.com)'s article
 "jcc7" <technocrat7 gmail.com> wrote in message
 news:f9f7oc$2ovj$1 digitalmars.com...
 <snip>
 It's been mentioned on the wiki page at
 http://www.prowiki.org/wiki4d/wiki.cgi?WindowsAPI, but I think you're
 right that
 it's not in the "Translation instructions" section (and it probably should
 be there).

 I get the feeling that the WindowsAPI page at Wiki4D has gotten way too
 big to be
 efficient. I think that it should probably be split into subpages, but I
 would be
 afraid of stepping on the toes of the WindowsAPI team by doing it myself.

approach.
 (Also, I wonder if it would work better to store the content that's
 current at
 Wiki4D in the bindings wiki instead
 [http://www.dsource.org/projects/bindings],
 since it is part of the bindings project and dsource wikis have a cool way
 of
 formatting D source code.)

Maybe that would be a good idea. I haven't used Dsource's wiki engine much - how does it compare to prowiki? At least it seems able to show arbitrary versions of a page like MediaWiki, rather than just the current version and the most recent edit.

If you click on the "Archive" link, it will show you more versions, but you're right that Trac does keep track of edits more thoroughly. As you noticed, Trac keeps all of the version of a file as it's changed (and it always keeps track of who changed it, too). On the other hand, I haven't quite been able to figure out how often ProWiki saves a version in the archive or even if they're saved permanently. And with ProWiki, it can be difficult to determine who changed what. Trac is really good at formatting D code automatically (for an example, see: http://www.dsource.org/projects/tutorials/wiki/PrintfExample). By the way, the old formatter that I had worked on has been replaced with a much cooler formatter that is Pygments-based. Trac allows you to get an RSS feed of project events (such when people change wiki pages and when SVN changes are committed). ProWiki will also give an RSS feed (which I think will even let you restrict your news to a particular page), but I think it's helpful to have the SVN changes and the wiki changes in the same feed. (I've been using RSS to watch that no one is being mischievous in the Tutorials project for a while.) Also, every wiki engine seems to have it's own flavor of wiki syntax, of course, but you probably already realized that. I think that using the Trac wiki pages would make more sense that using the ProWiki page. It probably would take a little time to change the pages from ProWiki syntax to Trac wiki syntax, but I don't think it'd be that bad. And you wouldn't have to change all of the content at once. You could move one section of the page, then delete it from ProWiki, and so on. I'd even help you if you want.
Aug 28 2007
prev sibling parent reply "Stewart Gordon" <smjg_1998 yahoo.com> writes:
"Michael Butscher" <mbutscher gmx.de> wrote in message 
news:MPG.212522dc134a28d69896a8 news.digitalmars.com...
<snip>
 This might be an indication that this part of the headers is currently
 not used widely (maybe by some people changing their copy of the
 headers privately). But the published version definitely can't work.

Which headers - std.c.windows or my project? And if people are creating private forks of my project, why? True, it's PD, so who's to stop them, but what's to gain by doing this instead of contributing to the project? <snip>
 I found out in the meantime that some headers (namely "d3d9.d") were
 modified already in a similar way, but it was not documented in the
 translation instructions.

The D3D9 headers have been completely retranslated from M$'s own headers by somebody who missed the point. Is MinGW 3.9 as up to date as this? I've just deassigned these modules - would somebody like to redo them from MinGW 3.9? <snip>
 Also, maybe this question is better suited in the Bindings project 
 Dsource forum?

You are right. I only found forums a bit impractical, but that's not a good argument.

And I don't tend to read the Dsource forums.... Stewart.
Aug 27 2007
parent Michael Butscher <mbutscher gmx.de> writes:
Stewart Gordon wrote:
 "Michael Butscher" <mbutscher gmx.de> wrote in message 
 news:MPG.212522dc134a28d69896a8 news.digitalmars.com...
 <snip>
 This might be an indication that this part of the headers is currently
 not used widely (maybe by some people changing their copy of the
 headers privately). But the published version definitely can't work.

Which headers - std.c.windows or my project?

I meant your headers. The std.c.windows headers work.
 And if people are creating private forks of my project, why?  True, it's PD, 
 so who's to stop them, but what's to gain by doing this instead of 
 contributing to the project?

Well, laziness is part of human nature. First one should ask if the changes made privately were done in the right way to not produce chaos in the repository. One has to find the subversion URL (I really had some problems with this step), check out the files, change it, check if the testcompile script works (which it didn't in the past due to, AFAIR d3d9.d) and finally commit the changes. [...]
 Also, maybe this question is better suited in the Bindings project 
 Dsource forum?

You are right. I only found forums a bit impractical, but that's not a good argument.

And I don't tend to read the Dsource forums....

With the bindings forum there is also the additional problem that it is used by multiple subprojects and one has to guess by subject line if the topic belongs to Windows bindings or something else. Michael
Sep 03 2007
prev sibling parent Don Clugston <dac nospam.com.au> writes:
Michael Butscher wrote:
 Hi,
 
 there seems to be a rather fundamental error in the translation of COM 
 interfaces by the Win32 bindings project:
 
 The C++ headers defines an interface as a class (more precisely a 
 struct) with some virtual methods. This means, an instance of the 
 interface is the actual object data, mainly containing a pointer to the 
 vtable.
 
 In D, an interface is a pointer to the object data which contains in 
 turn the pointer to the vtable.
 
 Because of the different level of indirection e.g. the IShellFolder 
 interface in D is in fact equivalent to the LPSHELLFOLDER type in C++.
 
 
 I think there are two possible solutions:
 
 1. Modify parameter types of all functions using the interface, e.g. 
 translate C++ definition:
 
   HRESULT SHGetDesktopFolder(LPSHELLFOLDER*);
 
 
 to:
   
   HRESULT SHGetDesktopFolder(LPSHELLFOLDER);
 
 or:
 
   HRESULT SHGetDesktopFolder(IShellFolder*);
 
 
 
 
 2. Rewrite the definitions of the LP... types as aliases of the 
 interfaces (I would prefer that), e.g.
 
 C++:
 
   typedef IShellFolder *LPSHELLFOLDER;
 
 
 to D:
 
   alias IShellFolder LPSHELLFOLDER;

I think you're right (and I'm probably responsible for most of it). It's a straightforward but fundamental error. Essentially, there's already a * in the definition of a D class. The second way is how it should be. Should be straightforward to fix with a simple bit of grepping.
Aug 09 2007