www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - The issue with libraries that use Windows API prototypes

reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
I've run into yet another incompatibility issue with D2 libraries that
use the Windows API.
This time it's Derelict:

testderelict.obj(testderelict)
 Error 42: Symbol Undefined _wglDeleteContext 4
testderelict.obj(testderelict)
 Error 42: Symbol Undefined _wglMakeCurrent 8
testderelict.obj(testderelict)
 Error 42: Symbol Undefined _wglCreateContext 4
--- errorlevel 3

The issue? Derelict uses its own Windows API header prototype in
\Derelict2\DerelictUtil\derelict\util, which defines:

alias void* HANDLE;
alias HANDLE HGLRC;

And the "_wgl" functions take HGLRC as the parameter.

The problem is WindowsAPI also defines these, because well, it's a
binding library so it has to. And it's prototypes are:

typedef void* HANDLE;
alias HANDLE HGLRC;

When linking my project with WinsowsAPI and Derelict, the Derelict
static library defines a function as "_wglDeleteContext", while the
project that uses WindowsAPI sees the wglDeleteContext prototype from
the import, but sees the HGLRC typedef in WindowsAPI, and DMD looks
for the symbol "_wglDeleteContext 4".

So now I can't really use Derelict and the WindowsAPI. Perhaps I can
use it with std.c.windows.windows? Nope, guess again!

std.c.windows.windows doesn't define NULL (small nuisance), and it
doesn't use aliases to ANSI/WideChar versions of API functions like
the WindowsAPI bindings, you have to explicitly use e.g. MessageBoxA
or MessageBoxW.

To add insult to injury, MessageBoxW isn't even defined in
\druntime\src\core\sys\windows\windows.d (that's imported by
std.c.windows.windows). It defines maybe 5% of anything in WindowsAPI,
so using std.c.windows.windows is out of the question because of an
enormous wall of errors due to missing symbols.

But even so, std.c.windows.windows *also* defines HANDLE:
typedef void *HANDLE;

So far, we have 3 incompatible libraries. Add a couple of more
libraries like DFL, DGui DCairo and now we're up to 6.

*Every single* Windows-related library seems to define its own winapi
prototypes, and because of issues like:
1. typedefs (which we're going to hopefully kill sometime soon)
2. small differences in how these prototypes were written

these libraries become incompatible with each other. It's only going
to get worse as each library author keeps growing their library and
adds more prototypes.

I can see two ways out of this situation:
1) D Library authors start using the WindowsAPI bindings from
http://dsource.org/projects/bindings/wiki/WindowsApi, and expect the
user of the their library to put WindowsAPI in the import path where
the authors' libraries can find it to resolve symbols, and where the
user's project can import it as well. This would mean the author's
library's build process gets a bit more complicated as you would have
to pass the path to the WindowsAPI bindings when building the author's
library.

2) We devise some plan to incorporate WindowsAPI into Phobos itself.
This way library users don't have to waste time fetching dependencies,
and library authors don't have to waste time prototyping functions
since the majority of this work will be already incorporated into
Phobos.

The DCairo author kindly suggested to use version(WindowsAPI)
else(HisOwnPrototypes) as a workaround, but this doesn't scale. There
could be numerous prototypes based on how many libraries you use.


somewhere because the more time we wait, the worse this situation
gets.
Jul 09 2011
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-07-09 18:55, Andrej Mitrovic wrote:

 I've run into yet another incompatibility issue with D2 libraries that
 use the Windows API.
 This time it's Derelict:

 testderelict.obj(testderelict)
   Error 42: Symbol Undefined _wglDeleteContext 4
 testderelict.obj(testderelict)
   Error 42: Symbol Undefined _wglMakeCurrent 8
 testderelict.obj(testderelict)
   Error 42: Symbol Undefined _wglCreateContext 4
 --- errorlevel 3

 The issue? Derelict uses its own Windows API header prototype in
 \Derelict2\DerelictUtil\derelict\util, which defines:

 alias void* HANDLE;
 alias HANDLE HGLRC;

 And the "_wgl" functions take HGLRC as the parameter.

 The problem is WindowsAPI also defines these, because well, it's a
 binding library so it has to. And it's prototypes are:

 typedef void* HANDLE;
 alias HANDLE HGLRC;

 When linking my project with WinsowsAPI and Derelict, the Derelict
 static library defines a function as "_wglDeleteContext", while the
 project that uses WindowsAPI sees the wglDeleteContext prototype from
 the import, but sees the HGLRC typedef in WindowsAPI, and DMD looks
 for the symbol "_wglDeleteContext 4".

 So now I can't really use Derelict and the WindowsAPI. Perhaps I can
 use it with std.c.windows.windows? Nope, guess again!

 std.c.windows.windows doesn't define NULL (small nuisance), and it
 doesn't use aliases to ANSI/WideChar versions of API functions like
 the WindowsAPI bindings, you have to explicitly use e.g. MessageBoxA
 or MessageBoxW.

 To add insult to injury, MessageBoxW isn't even defined in
 \druntime\src\core\sys\windows\windows.d (that's imported by
 std.c.windows.windows). It defines maybe 5% of anything in WindowsAPI,
 so using std.c.windows.windows is out of the question because of an
 enormous wall of errors due to missing symbols.

 But even so, std.c.windows.windows *also* defines HANDLE:
 typedef void *HANDLE;

 So far, we have 3 incompatible libraries. Add a couple of more
 libraries like DFL, DGui DCairo and now we're up to 6.


We can probably add DWT to this list as well. This seems like quite a 
mess that needs to be sorted out. It's hard to tell what the best 
solution would be. There need to be some kind of bindings in druntime, 
since it depends on the Windows APIs. On the other hand it seems to be 
unnecessary to have DirectX bindings and similar in druntime. Perhaps 
the best solution is to have an absolute minimum set of bindings in 
druntime and then merge all other bindings in the Windows API bindings 
project. This would be less of an issue when we get a package manager for D.

-- 
/Jacob Carlborg
Jul 10 2011
parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/10/11, Jacob Carlborg <doob me.com> wrote:

 This would be less of an issue when we get a package manager for D.


Yeah I would agree with this. For example, to use the winapi in Python
you would have to install http://sourceforge.net/projects/pywin32/ .
Although for Windows these python packages come in a installation
setup form, it's still damn easy to install one and even pick which
Python version to install it for. As soon as you've got that package
installed, you're pretty much covered for any python code that uses
winapi.

DSSS seemed like a good idea although its codebase scares me a little. :>
Jul 10 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-07-10 15:50, Andrej Mitrovic wrote:

 On 7/10/11, Jacob Carlborg<doob me.com>  wrote:

 This would be less of an issue when we get a package manager for D.


 Yeah I would agree with this. For example, to use the winapi in Python
 you would have to install http://sourceforge.net/projects/pywin32/ .
 Although for Windows these python packages come in a installation
 setup form, it's still damn easy to install one and even pick which
 Python version to install it for. As soon as you've got that package
 installed, you're pretty much covered for any python code that uses
 winapi.

 DSSS seemed like a good idea although its codebase scares me a little. :>


I'm already working on a package manager for D.

-- 
/Jacob Carlborg
Jul 10 2011
prev sibling next sibling parent Trass3r <un known.com> writes:
 2) We devise some plan to incorporate WindowsAPI into Phobos itself.
 This way library users don't have to waste time fetching dependencies,
 and library authors don't have to waste time prototyping functions
 since the majority of this work will be already incorporated into
 Phobos.


Long overdue indeed.
Jul 10 2011
prev sibling parent Stewart Gordon <smjg_1998 yahoo.com> writes:
On 09/07/2011 17:55, Andrej Mitrovic wrote:
<snip>

 alias void* HANDLE;
 alias HANDLE HGLRC;

 And the "_wgl" functions take HGLRC as the parameter.

 The problem is WindowsAPI also defines these, because well, it's a
 binding library so it has to. And it's prototypes are:

 typedef void* HANDLE;
 alias HANDLE HGLRC;


I don't know Derelict at all.  But if as I make out it's a cross-platform
library, I'm 
surprised if many functions in its API receive or return Windows API types
directly, which 
should be the only case in which this matters.  But for passing a HANDLE to a
Derelict 
function, it still shouldn't be a problem as a typedef can be implicitly
converted to its 
underlying type.

derelict.util.wintypes.HANDLE and win32.basetsd.HANDLE are distinct symbols. 
If your code 
uses WindowsAPI, Derelict can still use its own bindings internally.  Yes, it's
wasteful, 
but the code should still compile.


 When linking my project with WinsowsAPI and Derelict, the Derelict
 static library defines a function as "_wglDeleteContext", while the
 project that uses WindowsAPI sees the wglDeleteContext prototype from
 the import, but sees the HGLRC typedef in WindowsAPI, and DMD looks
 for the symbol "_wglDeleteContext 4".


And does it find that symbol?

If not, have you checked that you're linking in all the right .lib files?


 So now I can't really use Derelict and the WindowsAPI. Perhaps I can
 use it with std.c.windows.windows? Nope, guess again!

 std.c.windows.windows doesn't define NULL (small nuisance),


Why is it a nuisance at all?  D code should use null, which is built in, not
NULL.


 and it
 doesn't use aliases to ANSI/WideChar versions of API functions like
 the WindowsAPI bindings, you have to explicitly use e.g. MessageBoxA
 or MessageBoxW.


std.c.windows is very deficient (last time I knew, Walter just added stuff as
and when he 
needed it), so why single this omission out?


 *Every single* Windows-related library seems to define its own winapi
 prototypes, and because of issues like:
 1. typedefs (which we're going to hopefully kill sometime soon)
 2. small differences in how these prototypes were written

 these libraries become incompatible with each other. It's only going
 to get worse as each library author keeps growing their library and
 adds more prototypes.


It's inefficient, but since the symbols have distinct fully qualified names, I
can't see 
it causing clashes generally.  Trouble occurs if:

(a) A library API module (as opposed to a module for the library's internal
use) publicly 
defines one of these symbols or publicly imports a module that does, thereby
forcing you 
to use fully qualified names.

(b) You need to pass data of Windows API types directly between your app and
the library. 
  Then you end up needing to use casts.  You might also need to import a
library module 
that defines the types, thereby making (a) an issue here too.


 I can see two ways out of this situation:
 1) D Library authors start using the WindowsAPI bindings from
 http://dsource.org/projects/bindings/wiki/WindowsApi, and expect the
 user of the their library to put WindowsAPI in the import path where
 the authors' libraries can find it to resolve symbols, and where the
 user's project can import it as well. This would mean the author's
 library's build process gets a bit more complicated as you would have
 to pass the path to the WindowsAPI bindings when building the author's
 library.


Then tell D programmers to install library code under a common base path, so
that at most 
one extra import path in sc.ini is needed however many third party libraries
you use.


 2) We devise some plan to incorporate WindowsAPI into Phobos itself.
 This way library users don't have to waste time fetching dependencies,
 and library authors don't have to waste time prototyping functions
 since the majority of this work will be already incorporated into
 Phobos.


<snip>

http://d.puremagic.com/issues/show_bug.cgi?id=317

Moreover, it's about time we had a plan for getting these bindings completed. 
Unfortunately now that I have a job....

Stewart.
Jul 10 2011