www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Higher abstraction level for calling C functions

reply Denis Shelomovskij <verylonglogin.reg gmail.com> writes:
D has complete (IMHO) compiler support for calling C functions (using 
extern(C)). But there is a lack of library support.

Readers from this NG probably know this, but... Microsoft .NET Framework 
has such support. Look briefly at source example in:
http://msdn.microsoft.com/en-gb/library/s97shtze.aspx

And at System.IO.Directory.GetCurrentDirectory source:
http://typedescriptor.net/name/members/5540086E017CD13896E80A0CAEA6E517-System.IO.Directory.GetCurrentDirectory%28%29
where Win32Native.GetCurrentDirectory is defined as:
---
[DllImport("kernel32.dll", CharSet=CharSet.Auto, SetLastError=true)]
internal static extern int GetCurrentDirectory(int nBufferLength, 
StringBuilder lpBuffer);
---

Looks good? But it's not. It tries to provide both low level support 
(it's needed to just call C function, like D extern(C), but do 
everything at runtime (yes, in .NET Framework even Regex can be compiled 
at runtime from a string)) and high level support (it converts UTF-16 to 
ANSI in the given MSDN example). As a result it even doesn't allow you 
to specify a (scalar) parameter as a length of another (array) parameter 
or throw on native function failure.



What can be done in that direction in D? The whole my point is expressed 
in my small cwrap library:
http://deoma-cmd.ru/d/docs/src/cwrap.html

"Examples" is the most informative section of the documentation. Other 
examples can be found in (original functions, all in one file):
https://bitbucket.org/denis_sh/cwrap/src/tip/examples/c.d
and generated wrappers:
https://bitbucket.org/denis_sh/cwrap/src/tip/out

Library state: almost every internal function is unittested, generates 
compilable code which looks right but not unittested yet.
NOTE: You need head dmd to CTFE it or you can just change `enum data` to 
`auto data` in examples/c.d, this will also reduce compilation time more 
than x10.

Once original function is properly described in IDL, such library gives 
the fallowing advantages:
* [memory-corruption-safe] User works with arrays, not separate pointers 
and lengths.
* [memory-leak-safe] Original function allocated memory will be freed on 
any failure (NOTE: see "Original function restrictions" in documentation).
* [fail-safe] An exception will be thrown if original function fails.
* [usability] Much easier to work with arrays/preferred string types (it 
converts UTF-8/16/32 back and forth, see docs).
* [speed] No unnecessary GC (or other) allocations/copying/reading.
* [garbage-free] Look at, e.g. std.file - it leaves lots of garbage 
because of calling native platform functions. Generated wrappers don't 
even allocate GC memory (except for output parameters).

Looks like this functionality is usable for auto-generated bindings, at 
least a useful GtkD still leaks memory almost everywhere because it 
never frees (OK, not sure, I just haven't seen) allocated memory marked 
by "free with g_free".

So should such stuff be ever included in Phobos? IMHO yes because it's a 
general stuff. But if not, can generated wrappers only be included and 
used to improve current situation "a system language with a library 
which leaves garbage"?

By the way, std.file.getcwd and std.windows.charset.toMBSz are examples 
of hand-written wrappers (the first is mine and strictly speaking forced 
me to do this lib).

P.S.
Not sure if this library usable at all because I'm as sure in its 
usability as in tabs indention superiority over spaces indention. )
So if it's not (usable), tell me without scruple.


Thanks for reading!
Jan 29 2012
next sibling parent "Marco Leise" <Marco.Leise gmx.de> writes:
Am 29.01.2012, 21:48 Uhr, schrieb Denis Shelomovskij  
<verylonglogin.reg gmail.com>:

 D has complete (IMHO) compiler support for calling C functions (using  
 extern(C)). But there is a lack of library support.

 Readers from this NG probably know this, but... Microsoft .NET Framework  
 has such support. Look briefly at source example in:
 http://msdn.microsoft.com/en-gb/library/s97shtze.aspx

 And at System.IO.Directory.GetCurrentDirectory source:
 http://typedescriptor.net/name/members/5540086E017CD13896E80A0CAEA6E517-System.IO.Directory.GetCurrentDirectory%28%29
 where Win32Native.GetCurrentDirectory is defined as:
 ---
 [DllImport("kernel32.dll", CharSet=CharSet.Auto, SetLastError=true)]
 internal static extern int GetCurrentDirectory(int nBufferLength,  
 StringBuilder lpBuffer);
 ---

 Looks good? But it's not. It tries to provide both low level support  
 (it's needed to just call C function, like D extern(C), but do  
 everything at runtime (yes, in .NET Framework even Regex can be compiled  
 at runtime from a string)) and high level support (it converts UTF-16 to  
 ANSI in the given MSDN example). As a result it even doesn't allow you  
 to specify a (scalar) parameter as a length of another (array) parameter  
 or throw on native function failure.



 What can be done in that direction in D? The whole my point is expressed  
 in my small cwrap library:
 http://deoma-cmd.ru/d/docs/src/cwrap.html

 "Examples" is the most informative section of the documentation. Other  
 examples can be found in (original functions, all in one file):
 https://bitbucket.org/denis_sh/cwrap/src/tip/examples/c.d
 and generated wrappers:
 https://bitbucket.org/denis_sh/cwrap/src/tip/out

 Library state: almost every internal function is unittested, generates  
 compilable code which looks right but not unittested yet.
 NOTE: You need head dmd to CTFE it or you can just change `enum data` to  
 `auto data` in examples/c.d, this will also reduce compilation time more  
 than x10.

 Once original function is properly described in IDL, such library gives  
 the fallowing advantages:
 * [memory-corruption-safe] User works with arrays, not separate pointers  
 and lengths.
 * [memory-leak-safe] Original function allocated memory will be freed on  
 any failure (NOTE: see "Original function restrictions" in  
 documentation).
 * [fail-safe] An exception will be thrown if original function fails.
 * [usability] Much easier to work with arrays/preferred string types (it  
 converts UTF-8/16/32 back and forth, see docs).
 * [speed] No unnecessary GC (or other) allocations/copying/reading.
 * [garbage-free] Look at, e.g. std.file - it leaves lots of garbage  
 because of calling native platform functions. Generated wrappers don't  
 even allocate GC memory (except for output parameters).

 Looks like this functionality is usable for auto-generated bindings, at  
 least a useful GtkD still leaks memory almost everywhere because it  
 never frees (OK, not sure, I just haven't seen) allocated memory marked  
 by "free with g_free".

 So should such stuff be ever included in Phobos? IMHO yes because it's a  
 general stuff. But if not, can generated wrappers only be included and  
 used to improve current situation "a system language with a library  
 which leaves garbage"?

 By the way, std.file.getcwd and std.windows.charset.toMBSz are examples  
 of hand-written wrappers (the first is mine and strictly speaking forced  
 me to do this lib).

 P.S.
 Not sure if this library usable at all because I'm as sure in its  
 usability as in tabs indention superiority over spaces indention. )
 So if it's not (usable), tell me without scruple.


 Thanks for reading!
This idea came to me the other day when working with Windows and Posix API. Out of the enhancements you describe I find most useful: - string / array conversion - memory management - errno / return value to exception (already implemented as a function in exception.d) The only reason not to use your wrappers, is if they don't expose all features of the original or if you don't need D types because you pass the result of function A to function B directly. This could happen with some char* returning functions where you don't want to slow down your application by converting forth and back from Windows charset to UTF-8. It is definitely something for the "modern convenience" bullet point.
Jan 29 2012
prev sibling next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 01/29/2012 09:48 PM, Denis Shelomovskij wrote:
 D has complete (IMHO) compiler support for calling C functions (using
 extern(C)). But there is a lack of library support.

 Readers from this NG probably know this, but... Microsoft .NET Framework
 has such support. Look briefly at source example in:
 http://msdn.microsoft.com/en-gb/library/s97shtze.aspx

 And at System.IO.Directory.GetCurrentDirectory source:
 http://typedescriptor.net/name/members/5540086E017CD13896E80A0CAEA6E517-System.IO.Directory.GetCurrentDirectory%28%29

 where Win32Native.GetCurrentDirectory is defined as:
 ---
 [DllImport("kernel32.dll", CharSet=CharSet.Auto, SetLastError=true)]
 internal static extern int GetCurrentDirectory(int nBufferLength,
 StringBuilder lpBuffer);
 ---

 Looks good? But it's not. It tries to provide both low level support
 (it's needed to just call C function, like D extern(C), but do
 everything at runtime (yes, in .NET Framework even Regex can be compiled
 at runtime from a string)) and high level support (it converts UTF-16 to
 ANSI in the given MSDN example). As a result it even doesn't allow you
 to specify a (scalar) parameter as a length of another (array) parameter
 or throw on native function failure.



 What can be done in that direction in D? The whole my point is expressed
 in my small cwrap library:
 http://deoma-cmd.ru/d/docs/src/cwrap.html

 "Examples" is the most informative section of the documentation. Other
 examples can be found in (original functions, all in one file):
 https://bitbucket.org/denis_sh/cwrap/src/tip/examples/c.d
 and generated wrappers:
 https://bitbucket.org/denis_sh/cwrap/src/tip/out

 Library state: almost every internal function is unittested, generates
 compilable code which looks right but not unittested yet.
 NOTE: You need head dmd to CTFE it or you can just change `enum data` to
 `auto data` in examples/c.d, this will also reduce compilation time more
 than x10.

 Once original function is properly described in IDL, such library gives
 the fallowing advantages:
 * [memory-corruption-safe] User works with arrays, not separate pointers
 and lengths.
 * [memory-leak-safe] Original function allocated memory will be freed on
 any failure (NOTE: see "Original function restrictions" in documentation).
 * [fail-safe] An exception will be thrown if original function fails.
 * [usability] Much easier to work with arrays/preferred string types (it
 converts UTF-8/16/32 back and forth, see docs).
 * [speed] No unnecessary GC (or other) allocations/copying/reading.
 * [garbage-free] Look at, e.g. std.file - it leaves lots of garbage
 because of calling native platform functions. Generated wrappers don't
 even allocate GC memory (except for output parameters).

 Looks like this functionality is usable for auto-generated bindings, at
 least a useful GtkD still leaks memory almost everywhere because it
 never frees (OK, not sure, I just haven't seen) allocated memory marked
 by "free with g_free".

 So should such stuff be ever included in Phobos? IMHO yes because it's a
 general stuff.
+1.
 But if not, can generated wrappers only be included and
 used to improve current situation "a system language with a library
 which leaves garbage"?

 By the way, std.file.getcwd and std.windows.charset.toMBSz are examples
 of hand-written wrappers (the first is mine and strictly speaking forced
 me to do this lib).

 P.S.
 Not sure if this library usable at all because I'm as sure in its
 usability as in tabs indention superiority over spaces indention. )
 So if it's not (usable), tell me without scruple.


 Thanks for reading!
Looks like very cool stuff. In this example: // from cstring.h __(out) int memcmp( __(in[num] ) void* ptr1, __(in[+num]) void* ptr2, __(in ) size_t num ) __( success: true ); I suppose the unary + before the second 'num' is required to disambiguate from which array length 'num' will be deduced? Shouldn't this restriction be lifted? (Obviously, if the lengths have to match, both are fine.) In this example: // from WinBase.h __(out) DWORD GetCurrentDirectoryW( __(in) DWORD nBufferLength, __(out""[nBufferLength]->""[return]) LPWSTR lpBuffer ) __( repeat: return > nBufferLength ) __( success: return ); // NOTE: current directory can't have zero length Maybe the generated bindings should rather look like char[] winapi_GetCurrentDirectoryW();
Jan 29 2012
parent Denis Shelomovskij <verylonglogin.reg gmail.com> writes:
30.01.2012 4:42, Timon Gehr пишет:
 I suppose the unary + before the second 'num' is required to
 disambiguate from which array length 'num' will be deduced? Shouldn't
 this restriction be lifted? (Obviously, if the lengths have to match,
 both are fine.)
Yes, this restriction isn't necessary. I just was too lazy to add new asserts in such cases. Fill be fixed.
 In this example:

 // from WinBase.h
 __(out) DWORD GetCurrentDirectoryW(
 __(in) DWORD nBufferLength,
 __(out""[nBufferLength]->""[return]) LPWSTR lpBuffer
 )
 __( repeat: return > nBufferLength )
 __( success: return );
 // NOTE: current directory can't have zero length

 Maybe the generated bindings should rather look like

 char[] winapi_GetCurrentDirectoryW();
Maybe. Looks like "if return value is a length, corresponding array should be returned instead" rule is good. If I will not find anything major against this rule will be added.
Jan 30 2012
prev sibling parent reply "Kagamin" <spam here.lot> writes:
 static assert(__traits(compiles, HANDLE.init is null));
hmm... What declaration of HANDLE do you use?
Jan 29 2012
parent reply Denis Shelomovskij <verylonglogin.reg gmail.com> writes:
30.01.2012 8:59, Kagamin пишет:
 static assert(__traits(compiles, HANDLE.init is null));
hmm... What declaration of HANDLE do you use?
Wrapper knows nothing about non-standard types, but in `CreateFileW` the last argument is `__(in?) HANDLE hTemplateFile`, so it is an optional parameter (can be `null`) and wrapper ensures that it isn't a mistake and `HANDLE` is nullable.
Jan 30 2012
parent reply "Kagamin" <spam here.lot> writes:
On Monday, 30 January 2012 at 10:53:17 UTC, Denis Shelomovskij 
wrote:
 30.01.2012 8:59, Kagamin пишет:
 static assert(__traits(compiles, HANDLE.init is null));
hmm... What declaration of HANDLE do you use?
Wrapper knows nothing about non-standard types, but in `CreateFileW` the last argument is `__(in?) HANDLE hTemplateFile`, so it is an optional parameter (can be `null`) and wrapper ensures that it isn't a mistake and `HANDLE` is nullable.
Will it compile for alias int HANDLE; ?
Jan 30 2012
parent "Kagamin" <spam here.lot> writes:
On Monday, 30 January 2012 at 15:51:40 UTC, Kagamin wrote:
 On Monday, 30 January 2012 at 10:53:17 UTC, Denis Shelomovskij 
 wrote:
 30.01.2012 8:59, Kagamin пишет:
 static assert(__traits(compiles, HANDLE.init is null));
hmm... What declaration of HANDLE do you use?
Wrapper knows nothing about non-standard types, but in `CreateFileW` the last argument is `__(in?) HANDLE hTemplateFile`, so it is an optional parameter (can be `null`) and wrapper ensures that it isn't a mistake and `HANDLE` is nullable.
Will it compile for alias int HANDLE; ?
Ah, got it.
Jan 30 2012