www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Windows API and druntime/Phobos

reply Jonathan M Davis <jmdavisProg gmx.com> writes:
The level of support for the Windows API in druntime and Phobos is pretty low. 
As I understand it, Windows users are pretty much forced to use 
http://www.dsource.org/projects/bindings/browser/trunk/win32 if they need 
comprehensive Win32 API bindings. druntime seems to be trying to define all of 
the OS-specific stuff like that, but on top of it missing much of it, in the 
case of the Win32 API, that's a _lot_ of functions, and I don't know if we 
want to put that much in druntime. So, the question is, how do we want to 
support the Win32 API in druntime and Phobos?

Do we want to put all of the Win32 API bindings in druntime? If not, then do 
we want to put them in Phobos? Or do we just want to send Windows developers 
to a 3rd party library like the Win32 bindings project on dsource? Given that 
they're OS bindings, I would _think_ that we'd want them in druntime, but I 
don't know.

Regardless, this is one of those issues which frequently plagues D Windows 
developers, and we really should at least get a plan together as to how we 
want to handle it.

- Jonathan M Davis


P.S. A related pull request: https://github.com/D-Programming-
Language/druntime/pull/139
Jan 24 2012
next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Wednesday, 25 January 2012 at 19:16:34 UTC, Jonathan M Davis 
wrote:
 The level of support for the Windows API in druntime and Phobos 
 is pretty low. As I understand it, Windows users are pretty 
 much forced to use 
 http://www.dsource.org/projects/bindings/browser/trunk/win32 if 
 they need comprehensive Win32 API bindings. druntime seems to 
 be trying to define all of the OS-specific stuff like that, but 
 on top of it missing much of it, in the case of the Win32 API, 
 that's a _lot_ of functions, and I don't know if we want to put 
 that much in druntime. So, the question is, how do we want to 
 support the Win32 API in druntime and Phobos?

 Do we want to put all of the Win32 API bindings in druntime? If 
 not, then do we want to put them in Phobos? Or do we just want 
 to send Windows developers to a 3rd party library like the 
 Win32 bindings project on dsource? Given that they're OS 
 bindings, I would _think_ that we'd want them in druntime, but 
 I don't know.

I support putting all public domain modules from the Win32 bindings project into Druntime. The entirety of the Windows API modules (headers in C) and accompanying import libraries are an integral part of The Windows SDK. There is no obvious finer-grained separation of modules, so including only a part of them would need to be done on mostly subjective decisions. The only non-public-domain modules are the DirectX headers (which don't belong in Druntime either way), and WinSock modules, which are "Portions Copyright (c) 1980, 1983, 1988, 1993 The Regents of the University of California. All rights reserved." and "Portions Copyright (c) 1993 by Digital Equipment Corporation."
Jan 25 2012
prev sibling next sibling parent reply Don <nospam nospam.com> writes:
On 25.01.2012 05:48, Jonathan M Davis wrote:
 The level of support for the Windows API in druntime and Phobos is pretty low.
 As I understand it, Windows users are pretty much forced to use
 http://www.dsource.org/projects/bindings/browser/trunk/win32 if they need
 comprehensive Win32 API bindings. druntime seems to be trying to define all of
 the OS-specific stuff like that, but on top of it missing much of it, in the
 case of the Win32 API, that's a _lot_ of functions, and I don't know if we
 want to put that much in druntime. So, the question is, how do we want to
 support the Win32 API in druntime and Phobos?

 Do we want to put all of the Win32 API bindings in druntime? If not, then do
 we want to put them in Phobos? Or do we just want to send Windows developers
 to a 3rd party library like the Win32 bindings project on dsource? Given that
 they're OS bindings, I would _think_ that we'd want them in druntime, but I
 don't know.

 Regardless, this is one of those issues which frequently plagues D Windows
 developers, and we really should at least get a plan together as to how we
 want to handle it.

 - Jonathan M Davis


 P.S. A related pull request: https://github.com/D-Programming-
 Language/druntime/pull/139

Please note that the Win32 API project on dsource was created with the explicit intention of including it in Phobos. See bug 317.
Jan 25 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-01-26 00:50, Jonathan M Davis wrote:
 On Wednesday, January 25, 2012 20:38:01 Don wrote:
 Please note that the Win32 API project on dsource was created with the
 explicit intention of including it in Phobos. See bug 317.

Good to know. I would point out though that for the most part, std.c has been being phased out with all of its parts being moved to Phobos, so I'm not sure that putting the Win32 API bindings in std.c.windows makes sense. Regardless, my point is that we need to figure out what we want to do at this point. As it stands, we end up with pull requests such as https://github.com/D-Programming-Language/druntime/pull/139 which just add more bindings which a particular programmer happens to want. And while that might be okay in the short term, that doesn't work in the long term. We need to decide where we want the Win32 API bindings, how we want them organized more or less, and start putting them there. The Win32 API project is a great start, but it's been around for years without being moved into druntime or Phobos, and we should start looking at moving those bindings into either druntime or Phobos. At this point, given that they're OS bindings, and the OS bindings have been going in druntime, I would expect them to go into druntime. But a decision must be made, or the situation won't improve. - Jonathan M Davis

It depends. If we want to keep druntime small and only contain what's necessary for the runtime it would be better to add the bindings to Phobos. But that would mean to duplicate parts of the bindings. -- /Jacob Carlborg
Jan 25 2012
prev sibling next sibling parent "Jesse Phillips" <jessekphillips+D gmail.com> writes:
On Wednesday, 25 January 2012 at 19:16:34 UTC, Jonathan M Davis 
wrote:
 The level of support for the Windows API in druntime and Phobos 
 is pretty low. As I understand it, Windows users are pretty 
 much forced to use 
 http://www.dsource.org/projects/bindings/browser/trunk/win32 if 
 they need comprehensive Win32 API bindings. druntime seems to 
 be trying to define all of the OS-specific stuff like that, but 
 on top of it missing much of it, in the case of the Win32 API, 
 that's a _lot_ of functions, and I don't know if we want to put 
 that much in druntime. So, the question is, how do we want to 
 support the Win32 API in druntime and Phobos?

It seems to me that calling OS functions are expected to work out of the box. This would preclude requiring a third party library. I don't think it makes sense in Druntime. It isn't required for the compiler to work that I know of. Maybe their are some Windows functions used within druntime? Also what is there currently isn't generating documentation. Phobos already has std.windows, it just needs expanded.
Jan 25 2012
prev sibling next sibling parent "Yao Gómez" <yaoltzin gmail.com> writes:
On Wednesday, 25 January 2012 at 19:58:40 UTC, Jesse Phillips 
wrote:

[snip]
 Phobos already has std.windows, it just needs expanded.

The issue is that this would be a huge undertaking, as the std.windows (or core.sys.windows, to be more precise) is awfully incomplete. Maybe we could add the Win32 bindings into the Deimos project, and make it more prominent somehow. -- Yao Gómez
Jan 25 2012
prev sibling next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Wednesday, 25 January 2012 at 19:58:40 UTC, Jesse Phillips 
wrote:
 I don't think it makes sense in Druntime. It isn't required for 
 the compiler to work that I know of.

Underlying implementation of language features trickles down to using either the DMC runtime or Windows APIs. Druntime defines the Windows APIs locally where needed, which is a hack. Example: https://github.com/D-Programming-Language/druntime/blob/master/src/rt/dmain2.d#L31
 Also what is there currently isn't generating documentation.

This is not true: http://dlang.org/phobos/core_thread.html
Jan 25 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-01-26 00:44, Jonathan M Davis wrote:
 On Thursday, January 26, 2012 00:20:52 Jesse Phillips wrote:
 On Wednesday, 25 January 2012 at 21:44:21 UTC, Vladimir Panteleev

 wrote:
 Also what is there currently isn't generating documentation.

This is not true: http://dlang.org/phobos/core_thread.html

Yes it is, there aren't doc comments on anything: https://github.com/D-Programming-Language/druntime/blob/master/src/core/sys/ windows/windows.d

Pretty much none of the C bindings in druntime generate documentation - I believe primarily because Sean doesn't want to redo C documentation. However, that doesn't preclude giving them empty documentation so that you can see what's there, and that hasn't been done. - Jonathan M Davis

That would be good to have. A link to the C documentation would hurt either. -- /Jacob Carlborg
Jan 25 2012
prev sibling next sibling parent Trass3r <un known.com> writes:
 There is no obvious finer-grained separation of modules, so including  
 only a part of them would need to be done on mostly subjective decisions.

Yep, Walter just added what he used.
Jan 25 2012
prev sibling next sibling parent "Jesse Phillips" <jessekphillips+D gmail.com> writes:
On Wednesday, 25 January 2012 at 21:44:21 UTC, Vladimir Panteleev 
wrote:
 Also what is there currently isn't generating documentation.

This is not true: http://dlang.org/phobos/core_thread.html

Yes it is, there aren't doc comments on anything: https://github.com/D-Programming-Language/druntime/blob/master/src/core/sys/windows/windows.d
Jan 25 2012
prev sibling next sibling parent "Jesse Phillips" <jessekphillips+D gmail.com> writes:
On Wednesday, 25 January 2012 at 21:36:36 UTC, Yao Gómez wrote:
 On Wednesday, 25 January 2012 at 19:58:40 UTC, Jesse Phillips 
 wrote:

 [snip]
 Phobos already has std.windows, it just needs expanded.

The issue is that this would be a huge undertaking, as the std.windows (or core.sys.windows, to be more precise) is awfully incomplete. Maybe we could add the Win32 bindings into the Deimos project, and make it more prominent somehow. -- Yao Gómez

I don't see how that is the problem, this post is about what to do with the Win32 bindings project which is quite a bit more complete and created for the intent of inclusion of Phobos.
Jan 25 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, January 26, 2012 00:20:52 Jesse Phillips wrote:
 On Wednesday, 25 January 2012 at 21:44:21 UTC, Vladimir Panteleev
 
 wrote:
 Also what is there currently isn't generating documentation.

This is not true: http://dlang.org/phobos/core_thread.html

Yes it is, there aren't doc comments on anything: https://github.com/D-Programming-Language/druntime/blob/master/src/core/sys/ windows/windows.d

Pretty much none of the C bindings in druntime generate documentation - I believe primarily because Sean doesn't want to redo C documentation. However, that doesn't preclude giving them empty documentation so that you can see what's there, and that hasn't been done. - Jonathan M Davis
Jan 25 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, January 25, 2012 20:38:01 Don wrote:
 Please note that the Win32 API project on dsource was created with the
 explicit intention of including it in Phobos. See bug 317.

Good to know. I would point out though that for the most part, std.c has been being phased out with all of its parts being moved to Phobos, so I'm not sure that putting the Win32 API bindings in std.c.windows makes sense. Regardless, my point is that we need to figure out what we want to do at this point. As it stands, we end up with pull requests such as https://github.com/D-Programming-Language/druntime/pull/139 which just add more bindings which a particular programmer happens to want. And while that might be okay in the short term, that doesn't work in the long term. We need to decide where we want the Win32 API bindings, how we want them organized more or less, and start putting them there. The Win32 API project is a great start, but it's been around for years without being moved into druntime or Phobos, and we should start looking at moving those bindings into either druntime or Phobos. At this point, given that they're OS bindings, and the OS bindings have been going in druntime, I would expect them to go into druntime. But a decision must be made, or the situation won't improve. - Jonathan M Davis
Jan 25 2012
prev sibling next sibling parent "Kagamin" <spam here.lot> writes:
On Wednesday, 25 January 2012 at 19:27:47 UTC, Vladimir Panteleev 
wrote:
 I support putting all public domain modules from the Win32 
 bindings project into Druntime.

 The entirety of the Windows API modules (headers in C) and 
 accompanying import libraries are an integral part of The 
 Windows SDK.

Public domain? PSDK is not public domain. Isn't "Microsoft & public domain" an oxymoron?
Jan 27 2012
prev sibling next sibling parent Sean Kelly <sean invisibleduck.org> writes:
On Jan 25, 2012, at 11:27 AM, Vladimir Panteleev wrote:

 On Wednesday, 25 January 2012 at 19:16:34 UTC, Jonathan M Davis wrote:
 The level of support for the Windows API in druntime and Phobos is =


use http://www.dsource.org/projects/bindings/browser/trunk/win32 if they = need comprehensive Win32 API bindings. druntime seems to be trying to = define all of the OS-specific stuff like that, but on top of it missing = much of it, in the case of the Win32 API, that's a _lot_ of functions, = and I don't know if we want to put that much in druntime. So, the = question is, how do we want to support the Win32 API in druntime and = Phobos?
=20
 Do we want to put all of the Win32 API bindings in druntime? If not, =


Windows developers to a 3rd party library like the Win32 bindings = project on dsource? Given that they're OS bindings, I would _think_ that = we'd want them in druntime, but I don't know.
=20
 I support putting all public domain modules from the Win32 bindings =

=20
 The entirety of the Windows API modules (headers in C) and =

There is no obvious finer-grained separation of modules, so including = only a part of them would need to be done on mostly subjective = decisions.
=20
 The only non-public-domain modules are the DirectX headers (which =

"Portions Copyright (c) 1980, 1983, 1988, 1993 The Regents of the = University of California. All rights reserved." and "Portions Copyright = (c) 1993 by Digital Equipment Corporation." For what it's worth, I tried integrating these headers ages ago (into = Tango I believe) and stopped mostly because they were broken at the = time. I imagine they're sorted by now, and so long as there's little = maintenance effort on my part and there are no license concerns I'm all = for it. The only other thing to be aware of is that each module linked = into an executable incurs some overhead (500 bytes minimum, if I = remember correctly), so moving to these new Win32 headers will increase = the footprint of D apps on Windows. I imagine it's a worthwhile = tradeoff, but I did want to bring this up so no one is surprised later.=
Jan 27 2012
prev sibling next sibling parent Sean Kelly <sean invisibleduck.org> writes:
On Jan 25, 2012, at 11:58 AM, Jesse Phillips wrote:

 On Wednesday, 25 January 2012 at 19:16:34 UTC, Jonathan M Davis wrote:
 The level of support for the Windows API in druntime and Phobos is =


use http://www.dsource.org/projects/bindings/browser/trunk/win32 if they = need comprehensive Win32 API bindings. druntime seems to be trying to = define all of the OS-specific stuff like that, but on top of it missing = much of it, in the case of the Win32 API, that's a _lot_ of functions, = and I don't know if we want to put that much in druntime. So, the = question is, how do we want to support the Win32 API in druntime and = Phobos?
=20
 It seems to me that calling OS functions are expected to work out of =

=20
 I don't think it makes sense in Druntime. It isn't required for the =

used within druntime? Also what is there currently isn't generating = documentation. System headers are absolutely required for Druntime to build. This is = one of the reasons core.stdc and core.sys exist. Documentation isn't = generated because these are interfaces to external APIs, so the = documentation for those APIs should be referenced when coding to them. = We could add an empty doc comment to every declaration so something was = generated in the Druntime docs, but I don't know why anyone would ever = actually refer to them.=
Jan 27 2012
prev sibling next sibling parent Sean Kelly <sean invisibleduck.org> writes:
On Jan 25, 2012, at 1:44 PM, Vladimir Panteleev wrote:

 On Wednesday, 25 January 2012 at 19:58:40 UTC, Jesse Phillips wrote:
 I don't think it makes sense in Druntime. It isn't required for the =


=20
 Underlying implementation of language features trickles down to using =

APIs locally where needed, which is a hack.
=20
 Example:
=20
 =

n2.d#L31 This was done to reduce coupling between modules and to reduce the = runtime footprint in executables. It's not terribly = maintenance-friendly however, and so Druntime is slowly transition to = simply importing the appropriate module. A number of these remain = mostly because it ain't broke as-is.
Jan 27 2012
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, January 27, 2012 10:39:41 Sean Kelly wrote:
 System headers are absolutely required for Druntime to build. This is one
 of the reasons core.stdc and core.sys exist. Documentation isn't generated
 because these are interfaces to external APIs, so the documentation for
 those APIs should be referenced when coding to them. We could add an empty
 doc comment to every declaration so something was generated in the Druntime
 docs, but I don't know why anyone would ever actually refer to them.

The value in having empty ddoc comments is so that if you need to use any of the functions in them for some reason (which _does_ happen upon occasion), then you know what module has what without having to go find the source code and look through it. So, ideally, all of the declarations in druntime for the C APIs should at least have empty ddoc so that they're documented. I completely agree that it's best not to try and duplicate the full documentation for them - that's best left to the C docs - but documenting what's there is still valuable, and we should probably do that at some point. - Jonathan M Davis
Jan 27 2012
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 27 January 2012 at 18:16:38 UTC, Kagamin wrote:
 On Wednesday, 25 January 2012 at 19:27:47 UTC, Vladimir 
 Panteleev wrote:
 I support putting all public domain modules from the Win32 
 bindings project into Druntime.

 The entirety of the Windows API modules (headers in C) and 
 accompanying import libraries are an integral part of The 
 Windows SDK.

Public domain? PSDK is not public domain. Isn't "Microsoft & public domain" an oxymoron?

The files in the Windows API bindings project are translations from the MinGW headers, which were created based on the documentation (which explains the occasional errors). The MinGW headers are public domain. Considering that MinGW is not an obscure project, I think we're on safe legal ground here.
Jan 27 2012
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 27 January 2012 at 18:36:28 UTC, Sean Kelly wrote:
 The only other thing to be aware of is that each module linked 
 into an executable incurs some overhead (500 bytes minimum, if 
 I remember correctly), so moving to these new Win32 headers 
 will increase the footprint of D apps on Windows.  I imagine 
 it's a worthwhile tradeoff, but I did want to bring this up so 
 no one is surprised later.

Would it be possible to fix this overhead in the compiler (avoiding generation of such overhead for modules without classes, static ctors/dtors, etc.)? Even though Windows API programs classically begin with #include <windows.h>, which includes most of the other headers, this isn't a requirement, and it's possible to import modules on a finer-grained basis.
Jan 27 2012
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
I really hope druntime doesn't get one gigantic windows.d file. I
don't want to wait seconds for syntax highlighting to kick in because
a file is huge.
Jan 28 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, January 28, 2012 09:41:21 Andrej Mitrovic wrote:
 I really hope druntime doesn't get one gigantic windows.d file. I
 don't want to wait seconds for syntax highlighting to kick in because
 a file is huge.

It will definitely be more organized than that. I would think that it would be organized along the lines of how the win32 API itself is organized, but I don't know. It probably also depends on what the current bindings project does. A giant windows.d would definitely be rejected though. - Jonathan M Davis
Jan 28 2012
prev sibling next sibling parent "Kagamin" <spam here.lot> writes:
On Friday, 27 January 2012 at 20:37:04 UTC, Vladimir Panteleev 
wrote:
 The files in the Windows API bindings project are translations 
 from the MinGW headers, which were created based on the 
 documentation (which explains the occasional errors). The MinGW 
 headers are public domain. Considering that MinGW is not an 
 obscure project, I think we're on safe legal ground here.

Things like CRITICAL_SECTION are documented as opaque structures. http://social.msdn.microsoft.com/Forums/en-US/winembplatdev/thread/3856c437-9a7e-4315-a954-fc50e25a12f2
Jan 28 2012
prev sibling next sibling parent "Kagamin" <spam here.lot> writes:
I also remember MinGW folks were asked about this, but they don't 
know, where the headers came from.
Jan 28 2012
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Saturday, 28 January 2012 at 09:22:33 UTC, Jonathan M Davis 
wrote:
 On Saturday, January 28, 2012 09:41:21 Andrej Mitrovic wrote:
 I really hope druntime doesn't get one gigantic windows.d 
 file. I
 don't want to wait seconds for syntax highlighting to kick in 
 because
 a file is huge.

It will definitely be more organized than that. I would think that it would be organized along the lines of how the win32 API itself is organized, but I don't know. It probably also depends on what the current bindings project does. A giant windows.d would definitely be rejected though.

http://www.dsource.org/projects/bindings/browser/trunk/win32/windows.d
Jan 28 2012
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 1/28/12, Vladimir Panteleev <vladimir thecybershadow.net> wrote:
 http://www.dsource.org/projects/bindings/browser/trunk/win32/windows.d

Well sheeeit, somehow I never noticed that file being there.
Jan 28 2012
prev sibling next sibling parent "Marco Leise" <Marco.Leise gmx.de> writes:
Am 28.01.2012, 11:22 Uhr, schrieb Kagamin <spam here.lot>:

 I also remember MinGW folks were asked about this, but they don't know,  
 where the headers came from.

I understand that Microsoft prefers Visual Studio to be used with one of their compilers, but you can't honestly sue anyone nowadays for copying headers to interface with the OS. It is Windows, the most used OS on consumer PCs. Who's gain was it if a major part of open source software was locked out because they couldn't port the application due to missing OS headers. If you always looked only on the legal side of things, you would be allowed and disallowed absurd things: http://www.guardian.co.uk/politics/2007/nov/07/uk.queensspeech20072 The other ingredient is common sense. - So you don't buy Visual Studio, which may be a loss for MS, if those OSS developers would have switched to Windows, bought VS etc... in the other case, Windows loses popularity as a platform, because a lot of software wont be ported. With the background of the legal debate about making IE4 a part of the operating system I'd assume that some degree of freedom and choice of software is today considered almost as valuable as a formal copyright in a header file. It would be different if WINE somehow got the Windows source code and was only through this event becoming an alternative that made Windows obsolete. But I think we had the header/interface discussion here already :)
Jan 29 2012
prev sibling parent "Marco Leise" <Marco.Leise gmx.de> writes:
Am 28.01.2012, 09:41 Uhr, schrieb Andrej Mitrovic  
<andrej.mitrovich gmail.com>:

 I really hope druntime doesn't get one gigantic windows.d file. I
 don't want to wait seconds for syntax highlighting to kick in because
 a file is huge.

No, it should be done like everywhere else where the IDEs have no problem dealing with the Windows headers. C++, Delphi, ... . I think COM and WinSock are usually separate, whereas user32 and kernel32 are commonly imported through one file. I may be wrong. If a particular IDE/editor is slow it must be fixed :)
Jan 29 2012