www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - DStep - Bindings Generator 0.0.1

reply Jacob Carlborg <doob me.com> writes:
DStep is a tool for translating C and Objective-C headers to D modules. 
It uses libclang for lexing/parsing and AST traversal. This means it 
handles everything that Clang itself can handle, although this doesn't 
mean it will correctly translate everything.

I would consider this release alpha or beta. I'm releasing this now in 
hope I get some feedback on what language features the tool can't handle.

The tool is available at github:
https://github.com/jacob-carlborg/dstep

Binaries are available for Mac OS X and Ubuntu 11.10 32bit:
https://github.com/jacob-carlborg/dstep/downloads

Unfortunately I haven't been able to successfully compile it on Windows 
due to Optlink not cooperating. I'll most likely provide Linux binaries 
with better compatibility later.

Build instructions are available at github.

Usage:

dstep <input-file.h> -o output_file.d

For Objective-C

dstep <input-file.h> -o output_file.d -ObjC

Tests:

DStep uses Cucumber and Aruba (Ruby tools) to run its tests. It will 
basically run the tool on all *.h files in the "test_files" directory 
and compare the results to the corresponding *.d files.

Known issues/missing functionality:

* Multiple input files
* Framework as input file
* Add module declaration
* Option for specifying before and after code
* Option for specifying package
* Windows support

C:
     * Self includes
     * Out of order typedefs of structs
     * Bitfields
     * Non-standard extensions
     * Preprocessor
     * Arrays with no size marked as "extern".

Objective-C:
     * Protocols
     * Properties
     * Blocks
     * Categories
     * Actions
     * Outlets
     * Selectors

This is basically what's on the todo list:

https://raw.github.com/jacob-carlborg/dstep/master/todo.taskpaper

There's no point in reporting issues which are listed above.

-- 
/Jacob Carlborg
Jul 07 2012
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/7/2012 7:47 AM, Jacob Carlborg wrote:

 DStep is a tool for translating C and Objective-C headers to D modules. It uses
 libclang for lexing/parsing and AST traversal. This means it handles everything
 that Clang itself can handle, although this doesn't mean it will correctly
 translate everything.

 I would consider this release alpha or beta. I'm releasing this now in hope I
 get some feedback on what language features the tool can't handle.



I think this is potentially a big deal.

If it can be made complete enough, I'd like to add support into D for it, so
you 
could do things like:

     import "stdio.h";

and the D compile would fork/exec Dstep, generate the corresponding .d file,
and 
import the .d file.

Some issues:

1. Passing macro definitions to Dstep

2. The name "Dstep" has no obvious relationship to what it does.

3. The -o flag is not necessary. Just "do the right thing" when you see the 
filename extension.


In fact, we could make it a general facility, where if D sees:

     import "filename.ext";

that it fork/exec's the program:

     ext_to_D filename.ext tmpfile.d

and them imports tmpfile.d.
Jul 07 2012
next sibling parent "so" <so so.so> writes:
Wow at long last!

On Saturday, 7 July 2012 at 21:20:53 UTC, Walter Bright wrote:

 On 7/7/2012 7:47 AM, Jacob Carlborg wrote:

 DStep is a tool for translating C and Objective-C headers to D 
 modules. It uses
 libclang for lexing/parsing and AST traversal. This means it 
 handles everything
 that Clang itself can handle, although this doesn't mean it 
 will correctly
 translate everything.

 I would consider this release alpha or beta. I'm releasing 
 this now in hope I
 get some feedback on what language features the tool can't 
 handle.



 I think this is potentially a big deal.

 If it can be made complete enough, I'd like to add support into 
 D for it, so you could do things like:

     import "stdio.h";

 and the D compile would fork/exec Dstep, generate the 
 corresponding .d file, and import the .d file.

 Some issues:

 1. Passing macro definitions to Dstep

 2. The name "Dstep" has no obvious relationship to what it does.

 3. The -o flag is not necessary. Just "do the right thing" when 
 you see the filename extension.


 In fact, we could make it a general facility, where if D sees:

     import "filename.ext";

 that it fork/exec's the program:

     ext_to_D filename.ext tmpfile.d

 and them imports tmpfile.d.
Jul 07 2012
prev sibling next sibling parent Artur Skawina <art.08.09 gmail.com> writes:
On 07/07/12 23:20, Walter Bright wrote:

 In fact, we could make it a general facility, where if D sees:
 
     import "filename.ext";
 
 that it fork/exec's the program:
 
     ext_to_D filename.ext tmpfile.d
 
 and them imports tmpfile.d.



   import extern (C) "stdio.h";

which execs 

   dimport_C "stdio.h"

which returns a filename of a D module, which is then imported.

This way the importer can cache the result and doesn't need to
regenerate the D module until "stdio.h" or any dependency changes.


artur
Jul 07 2012
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/7/12 5:20 PM, Walter Bright wrote:

 In fact, we could make it a general facility, where if D sees:

 import "filename.ext";

 that it fork/exec's the program:

 ext_to_D filename.ext tmpfile.d

 and them imports tmpfile.d.



(Aside) This has an obvious security risk.

Andrei
Jul 07 2012
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 7/7/2012 7:59 PM, Andrei Alexandrescu wrote:

 On 7/7/12 5:20 PM, Walter Bright wrote:

 In fact, we could make it a general facility, where if D sees:

 import "filename.ext";

 that it fork/exec's the program:

 ext_to_D filename.ext tmpfile.d

 and them imports tmpfile.d.



 (Aside) This has an obvious security risk.



Yup. It's similar to the import arbitrary file as string feature. We can leave 
it disabled unless -J is used.
Jul 08 2012
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-07-08 04:59, Andrei Alexandrescu wrote:

 On 7/7/12 5:20 PM, Walter Bright wrote:

 In fact, we could make it a general facility, where if D sees:

 import "filename.ext";

 that it fork/exec's the program:

 ext_to_D filename.ext tmpfile.d

 and them imports tmpfile.d.



 (Aside) This has an obvious security risk.

 Andrei



Don't know if it helps but the tool can be easily used as a library and 
wrapped in a C API. Then DMD can used the tool directory like a library. 
No need for creating a new process and writing temporary files.

-- 
/Jacob Carlborg
Jul 08 2012
parent Walter Bright <newshound2 digitalmars.com> writes:
On 7/8/2012 4:15 AM, Jacob Carlborg wrote:

 Don't know if it helps but the tool can be easily used as a library and wrapped
 in a C API. Then DMD can used the tool directory like a library. No need for
 creating a new process and writing temporary files.



Creating a new process has its advantages:

1. Bugs from one process won't propagate to another.

2. The programmer of that process can develop completely independently of the D 
compiler.
Jul 08 2012
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-07-07 23:20, Walter Bright wrote:


 I think this is potentially a big deal.

 If it can be made complete enough, I'd like to add support into D for
 it, so you could do things like:

      import "stdio.h";

 and the D compile would fork/exec Dstep, generate the corresponding .d
 file, and import the .d file.



How is this going to work, is it going to be an optional feature? I 
mean, this will add DStep (D and Clang) as dependencies to DMD.

DStep is built to be used as a library, I can easily create a C API 
which can be used directly by DMD. No need for creating a new process. I 
can also make DStep give back the translate D code, no need for creating 
temporarily D files.

BTW, how would you indicate that the header file is an Objective-C file? 
Since both C and Objective-C uses the same extension for header files, 
this is required by Clang, otherwise it will treat the file as a C file.


 Some issues:

 1. Passing macro definitions to Dstep



Yeah, this will be the hardest. The main problem now is that libclang 
(the stable C API) doesn't have an API for handling macros.


 2. The name "Dstep" has no obvious relationship to what it does.



No it does not, but I'm tired of trying to come up with cleaver names 
for tools and libraries.

Here's the story behind the name for those who are interested:

DStep started out as another project, as a D-Objective-C bridge:

http://dsource.org/projects/dstep

In that project I had a tool for converting C/Objective-c headers to D 
modules. This tool was a Ruby script based on BridgeSupport. This is a 
complete rewrite of that tool. The whole project was called DStep and 
the name fit among other Objective-C related names like NeXTSTEP, 
OpenStep and GNUStep.


 3. The -o flag is not necessary. Just "do the right thing" when you see
 the filename extension.



That can be easily fixed.


 In fact, we could make it a general facility, where if D sees:

      import "filename.ext";

 that it fork/exec's the program:

      ext_to_D filename.ext tmpfile.d

 and them imports tmpfile.d.




-- 
/Jacob Carlborg
Jul 08 2012
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/8/2012 4:13 AM, Jacob Carlborg wrote:

 How is this going to work, is it going to be an optional feature? I mean, this
 will add DStep (D and Clang) as dependencies to DMD.



I think that implicitly using the feature will depend on those programs being 
available. It also means that any 3rd party can supply such a feature, to
import 
a file in any format.



 DStep is built to be used as a library, I can easily create a C API which can
be
 used directly by DMD. No need for creating a new process. I can also make DStep
 give back the translate D code, no need for creating temporarily D files.



I think there are many advantages to DStep being a separate program, not the 
least of which is debugging the output of it. Also, it means DStep could be 
written in any language. For example, suppose a Go-to-D is proposed. Go
provides 
a Go library to parse Go code - so such a tool might be more easily written in 
Go than in D.



 BTW, how would you indicate that the header file is an Objective-C file? Since
 both C and Objective-C uses the same extension for header files, this is
 required by Clang, otherwise it will treat the file as a C file.



Since OC is a proper superset of C, this shouldn't be a problem. Just run the
OC 
converter as your "C" compiler.



 In that project I had a tool for converting C/Objective-c headers to D modules.
 This tool was a Ruby script based on BridgeSupport. This is a complete rewrite
 of that tool. The whole project was called DStep and the name fit among other
 Objective-C related names like NeXTSTEP, OpenStep and GNUStep.



The name makes more sense now, but for marketing reasons it should give more of 
a clue as to what it does.
Jul 08 2012
parent reply Jacob Carlborg <doob me.com> writes:
On 2012-07-08 20:42, Walter Bright wrote:


 I think that implicitly using the feature will depend on those programs
 being available. It also means that any 3rd party can supply such a
 feature, to import a file in any format.



I see.


 I think there are many advantages to DStep being a separate program, not
 the least of which is debugging the output of it. Also, it means DStep
 could be written in any language. For example, suppose a Go-to-D is
 proposed. Go provides a Go library to parse Go code - so such a tool
 might be more easily written in Go than in D.



DStep is both usable as a program and a library. Both I guess you have a 
point. On the other hand, C can be used to glue together libraries 
written in different languages.


 Since OC is a proper superset of C, this shouldn't be a problem. Just
 run the OC converter as your "C" compiler.



That is not completely true if one is picky. The following code is legal 
C, but not legal Objective-C :

int id;
int nil;

In Objective-C "id" is a type.


 The name makes more sense now, but for marketing reasons it should give
 more of a clue as to what it does.



Do you have a suggestion?

-- 
/Jacob Carlborg
Jul 08 2012
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 09-Jul-12 02:13, Roman D. Boiko wrote:

 On Sunday, 8 July 2012 at 20:01:07 UTC, Jacob Carlborg wrote:

 On 2012-07-08 20:42, Walter Bright wrote:

 Since OC is a proper superset of C, this shouldn't be a problem. Just
 run the OC converter as your "C" compiler.



 That is not completely true if one is picky. The following code is
 legal C, but not legal Objective-C :

 int id;
 int nil;

 In Objective-C "id" is a type.



 particular file extension there may be several code importers. An
 (exotic?) example might be when some existing code uses one converter,
 but for some reason new code should use a different one.

 What about using the something like this:

 mixin convertImport!"header.h";



Ineffective even in distant future. Fixed functionality (=compiled, 
native, etc.) is faster and more practical.
E.g. the above was possible already for something like a year (no less)
the exact magic is:
mixin(translate(import("file.ext"));

But it never scaled to reasonably sized inputs/amounts of files like 
translating headers.


 with ability to specify a particular converter as second template
 parameter?



However something like :
import "file.ext", FancyImporter; could work and call some 
'FancyImporter' for compiler's tools directory to produce file.di

I think extra syntax could be added easily WHEN the need arrives, so far 
1:1 converters to extension feels fine.

-- 
Dmitry Olshansky
Jul 08 2012
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/8/2012 3:27 PM, Dmitry Olshansky wrote:

 I think extra syntax could be added easily WHEN the need arrives, so far 1:1
 converters to extension feels fine.



I don't think CTFE is good enough to parse C code in all its complex glory -
and 
it would be 1000 times too slow if it did.
Jul 08 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/8/12 9:39 PM, Walter Bright wrote:

 On 7/8/2012 3:27 PM, Dmitry Olshansky wrote:

 I think extra syntax could be added easily WHEN the need arrives, so
 far 1:1
 converters to extension feels fine.



 I don't think CTFE is good enough to parse C code in all its complex
 glory



disagree

  - and it would be 1000 times too slow if it did.

agree (for now)


Andrei
Jul 08 2012
prev sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Sunday, July 08, 2012 18:39:54 Walter Bright wrote:

 On 7/8/2012 3:27 PM, Dmitry Olshansky wrote:

 I think extra syntax could be added easily WHEN the need arrives, so far
 1:1 converters to extension feels fine.



 I don't think CTFE is good enough to parse C code in all its complex glory -
 and it would be 1000 times too slow if it did.



Unless you need to get at an external program or file, I would fully expect to 
be able to write a fully functional C parser using only CTFE-usable 
constructs. However, I expect that it would be disgusting to do so, and as you 
say, it would be horribly inefficient.

- Jonathan M Davis
Jul 08 2012
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/8/2012 1:01 PM, Jacob Carlborg wrote:

 That is not completely true if one is picky. The following code is legal C, but
 not legal Objective-C :

 int id;
 int nil;

 In Objective-C "id" is a type.



What do people do in OC makefiles?




 The name makes more sense now, but for marketing reasons it should give
 more of a clue as to what it does.



 Do you have a suggestion?



CtoD ?
Jul 08 2012
parent reply Jacob Carlborg <doob me.com> writes:
 What do people do in OC makefiles?



With Clang you can use the -ObjC or "-x objective-c" flags. But I guess 
most people use Xcode and not makefiles.


 CtoD ?



I'll have to think about it.

-- 
/Jacob Carlborg
Jul 08 2012
parent reply travert phare.normalesup.org (Christophe Travert) writes:
Jacob Carlborg , dans le message (digitalmars.D.announce:23893), a
 écrit :

 

 What do people do in OC makefiles?



 With Clang you can use the -ObjC or "-x objective-c" flags. But I guess 
 most people use Xcode and not makefiles.
 

 CtoD ?



 I'll have to think about it.
 
 -- 
 /Jacob Carlborg



CtoDi ? I suppose the tool translates interface only, not the code.
Jul 09 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-07-09 09:27, Christophe Travert wrote:


 CtoDi ? I suppose the tool translates interface only, not the code.



Yes, only declarations.

-- 
/Jacob Carlborg
Jul 09 2012
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/8/12 3:05 PM, Leandro Lucarella wrote:

 Jacob Carlborg, el  8 de July a las 13:13 me escribiste:

 On 2012-07-07 23:20, Walter Bright wrote:


 I think this is potentially a big deal.

 If it can be made complete enough, I'd like to add support into D for
 it, so you could do things like:

      import "stdio.h";

 and the D compile would fork/exec Dstep, generate the corresponding .d
 file, and import the .d file.



 How is this going to work, is it going to be an optional feature? I
 mean, this will add DStep (D and Clang) as dependencies to DMD.

 DStep is built to be used as a library, I can easily create a C API
 which can be used directly by DMD. No need for creating a new
 process. I can also make DStep give back the translate D code, no
 need for creating temporarily D files.



 This can also be done by just dumping the generated code to stdout.
 Then any application can use the output without using any temporary
 files. Pipes for the win!



Good old date checks (makefile style) can be used for the win.

Andrei
Jul 08 2012
prev sibling parent "Andrea Fontana" <nospam example.com> writes:
On Monday, 9 July 2012 at 07:27:35 UTC, 
travert phare.normalesup.org (Christophe Travert) wrote:

 Jacob Carlborg , dans le message 
 (digitalmars.D.announce:23893), a
  écrit :

 

 What do people do in OC makefiles?



 With Clang you can use the -ObjC or "-x objective-c" flags. 
 But I guess most people use Xcode and not makefiles.
 

 CtoD ?



 I'll have to think about it.
 
 --
 /Jacob Carlborg



 CtoDi ? I suppose the tool translates interface only, not the 
 code.



If I were him, I would choose "cited"
Jul 09 2012
prev sibling next sibling parent "Roman D. Boiko" <rb d-coding.com> writes:
On Sunday, 8 July 2012 at 20:01:07 UTC, Jacob Carlborg wrote:

 On 2012-07-08 20:42, Walter Bright wrote:

 Since OC is a proper superset of C, this shouldn't be a 
 problem. Just
 run the OC converter as your "C" compiler.



 That is not completely true if one is picky. The following code 
 is legal C, but not legal Objective-C :

 int id;
 int nil;

 In Objective-C "id" is a type.



particular file extension there may be several code importers. An 
(exotic?) example might be when some existing code uses one 
converter, but for some reason new code should use a different 
one.

What about using the something like this:

mixin convertImport!"header.h";

with ability to specify a particular converter as second template 
parameter?
Jul 08 2012
prev sibling next sibling parent "Roman D. Boiko" <rb d-coding.com> writes:
On Sunday, 8 July 2012 at 22:13:27 UTC, Roman D. Boiko wrote:

 What about using the something like this:

 mixin convertImport!"header.h";

 with ability to specify a particular converter as second 
 template parameter?



Oh, completely forgot to mention that inside mixin there could be 
a pragma for compiler instructing it to execute conversion.
Jul 08 2012
prev sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Saturday, 7 July 2012 at 21:20:53 UTC, Walter Bright wrote:

 If it can be made complete enough, I'd like to add support into 
 D for it, so you could do things like:

     import "stdio.h";



I don't think this syntax makes it clear enough. The following 
has been making rounds in the community for a long time:

     import(C) "stdio.h";
Jul 10 2012
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/7/12 10:47 AM, Jacob Carlborg wrote:

 DStep is a tool for translating C and Objective-C headers to D modules.



Awesome! On reddit:

http://www.reddit.com/r/programming/comments/w7hbg/dstep_tool_for_translating_c_and_objc_headers/


Andrei
Jul 07 2012
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 7/7/2012 8:40 PM, Andrei Alexandrescu wrote:

 http://www.reddit.com/r/programming/comments/w7hbg/dstep_tool_for_translating_c_and_objc_headers/




Gotta change the name:


http://www.reddit.com/r/programming/comments/w7hbg/dstep_tool_for_translating_c_and_objc_headers/c5az51y
Jul 08 2012
prev sibling next sibling parent "Bernard Helyer" <b.helyer gmail.com> writes:
On Sunday, 8 July 2012 at 08:36:34 UTC, Walter Bright wrote:

 On 7/7/2012 8:40 PM, Andrei Alexandrescu wrote:

 http://www.reddit.com/r/programming/comments/w7hbg/dstep_tool_for_translating_c_and_objc_headers/




 Gotta change the name:


 http://www.reddit.com/r/programming/comments/w7hbg/dstep_tool_for_translating_c_and_objc_headers/c5az51y



Just make it drop when you're done translating the file.
Jul 08 2012
prev sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Sunday, July 08, 2012 01:36:14 Walter Bright wrote:

 On 7/7/2012 8:40 PM, Andrei Alexandrescu wrote:

 http://www.reddit.com/r/programming/comments/w7hbg/dstep_tool_for_translat
 ing_c_and_objc_headers/






c2d?

- Jonathan M Davis
Jul 08 2012
prev sibling next sibling parent Leandro Lucarella <luca llucax.com.ar> writes:
Jacob Carlborg, el  8 de July a las 13:13 me escribiste:

 On 2012-07-07 23:20, Walter Bright wrote:


I think this is potentially a big deal.

If it can be made complete enough, I'd like to add support into D for
it, so you could do things like:

     import "stdio.h";

and the D compile would fork/exec Dstep, generate the corresponding .d
file, and import the .d file.



 How is this going to work, is it going to be an optional feature? I
 mean, this will add DStep (D and Clang) as dependencies to DMD.

 DStep is built to be used as a library, I can easily create a C API
 which can be used directly by DMD. No need for creating a new
 process. I can also make DStep give back the translate D code, no
 need for creating temporarily D files.



This can also be done by just dumping the generated code to stdout.
Then any application can use the output without using any temporary
files. Pipes for the win!

--
Leandro Lucarella (AKA luca)                     http://llucax.com.ar/
----------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145  104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------
Me encanta el éxito; por eso prefiero el estado de progreso constante,
con la meta al frente y no atrás.
	-- Ricardo Vaporeso. Punta del Este, Enero de 1918.
Jul 08 2012
prev sibling next sibling parent reply "Jonathan Andrew" <jonfandrew gmail.com> writes:
On Saturday, 7 July 2012 at 14:47:49 UTC, Jacob Carlborg wrote:

 DStep is a tool for translating C and Objective-C headers to D 
 modules. It uses libclang for lexing/parsing and AST traversal. 
 This means it handles everything that Clang itself can handle, 
 although this doesn't mean it will correctly translate 
 everything.

 I would consider this release alpha or beta. I'm releasing this 
 now in hope I get some feedback on what language features the 
 tool can't handle.

 The tool is available at github:
 https://github.com/jacob-carlborg/dstep

 Binaries are available for Mac OS X and Ubuntu 11.10 32bit:
 https://github.com/jacob-carlborg/dstep/downloads

 Unfortunately I haven't been able to successfully compile it on 
 Windows due to Optlink not cooperating. I'll most likely 
 provide Linux binaries with better compatibility later.

 Build instructions are available at github.

 Usage:

 dstep <input-file.h> -o output_file.d

 For Objective-C

 dstep <input-file.h> -o output_file.d -ObjC

 Tests:

 DStep uses Cucumber and Aruba (Ruby tools) to run its tests. It 
 will basically run the tool on all *.h files in the 
 "test_files" directory and compare the results to the 
 corresponding *.d files.

 Known issues/missing functionality:

 * Multiple input files
 * Framework as input file
 * Add module declaration
 * Option for specifying before and after code
 * Option for specifying package
 * Windows support

 C:
     * Self includes
     * Out of order typedefs of structs
     * Bitfields
     * Non-standard extensions
     * Preprocessor
     * Arrays with no size marked as "extern".

 Objective-C:
     * Protocols
     * Properties
     * Blocks
     * Categories
     * Actions
     * Outlets
     * Selectors

 This is basically what's on the todo list:

 https://raw.github.com/jacob-carlborg/dstep/master/todo.taskpaper

 There's no point in reporting issues which are listed above.



Jacob,

   I just used your tool to try and convert the GTK 3.0 include 
files into D stubs, and wanted to say that it worked pretty darn 
well. As a matter of practice, the single file limitation wasn't 
too much of a problem - I hacked together a script to just go 
through all the .h files and put the outputs in the right place. 
D has become my new favorite scripting language, BTW!

For anybody who cares - here's how I went about it. This is 
probably the most ignorant and least efficient method possible, 
but it worked for me to at least display a little window (yay.).

#!/usr/bin/rdmd

import std.stdio;
import std.string;
import std.process;

int main(string[] args)
{	
	foreach(string word; args[1..$])
	{
string cmd = format("dstep /home/jon/devel/gtk-include/%s -o 
/home/jon/devel/gtk-include/d/%s.d -v -I./ -I./glib-2.0/ -I./gdk/ 
-I./gdk-pixbuf/ -I/usr/lib/gcc/i686-linux-gnu/4.6/include/ 
-I/usr/include/pango-1.0/ -I/usr/include/cairo/ 
-I/usr/include/atk-1.0/", word, chomp(word,".h"));
		writefln("%s",cmd);
		system(cmd);
	}
	return 0;
}

Run with ./convert.d folder/*.h

The only disadvantage to the single-file limitation is that in 
the case of GTK at least, it has preprocessor directives to keep 
you from just #include-ing the single file you want to convert, 
so I just used sed to strip out all the #error directives that 
come up and force it to do my bidding. I understand DStep doesn't 
deal with preprocessor yet, but as far as the CLang front-end it 
uses goes, it might be helpful to find a way to turn off #error-s.

sed -i 's/#error/\/\//g' *.h

The next step was to rename all the D reserved words that GTK 
used as function arguments - in, out, function, and align are the 
only ones I can think of off the top of my head. Easy fix for the 
user (by no means am I complaining), but if you want to 
streamline the conversion, automatically renaming these kinds of 
arguments might be a helpful option.

Then, renaming all the duplicate empty struct{} entries in some 
of the files. You already know about this, but it was probably 
the most time-consuming part of the process for converting GTK, 
at least. I couldn't think of an easy way to automate this on my 
end, because some of the empty structs were necessary to get it 
to compile.

Finally, putting import statements in all the .d files after I 
was done. Still a long way to go on this (500 files).

Sorry for the long post, this is probably obvious stuff to 
everybody else, but I was really impressed with DStep - thank you 
for creating it!

-Jon
Jul 08 2012
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-07-08 23:22, Jonathan Andrew wrote:


 Jacob,




 The only disadvantage to the single-file limitation is that in the case
 of GTK at least, it has preprocessor directives to keep you from just
 #include-ing the single file you want to convert, so I just used sed to
 strip out all the #error directives that come up and force it to do my
 bidding. I understand DStep doesn't deal with preprocessor yet, but as
 far as the CLang front-end it uses goes, it might be helpful to find a
 way to turn off #error-s.



I had no idea about that.


 sed -i 's/#error/\/\//g' *.h

 The next step was to rename all the D reserved words that GTK used as
 function arguments - in, out, function, and align are the only ones I
 can think of off the top of my head. Easy fix for the user (by no means
 am I complaining), but if you want to streamline the conversion,
 automatically renaming these kinds of arguments might be a helpful option.



I thought the tool did that already.


 Then, renaming all the duplicate empty struct{} entries in some of the
 files. You already know about this, but it was probably the most
 time-consuming part of the process for converting GTK, at least. I
 couldn't think of an easy way to automate this on my end, because some
 of the empty structs were necessary to get it to compile.



I thought I had fixed this too. I'll have to take a look.


 Finally, putting import statements in all the .d files after I was done.
 Still a long way to go on this (500 files).

 Sorry for the long post, this is probably obvious stuff to everybody
 else, but I was really impressed with DStep - thank you for creating it!



No it's good, this is just what I wanted people to do. It would be great 
if you could report these issues:

https://github.com/jacob-carlborg/dstep/issues

If you have a simple test case or a header I can try that would be great.

-- 
/Jacob Carlborg
Jul 08 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-07-11 02:38, Jonathan Andrew wrote:


 OK, as far as the empty struct-s, it looks like it has to do with
 typedef struct.
 -------------------------------------------------------
 //Test.h:

 typedef struct _Booger Booger;

 //Results in:

 -------------------------------------------------------
 //Test.d:

 extern (C):

 alias _Booger Booger;

 struct _Booger
 {
 }



Reported as: https://github.com/jacob-carlborg/dstep/issues/4


 If the .h has:

 typedef struct _Booger Booger;

 struct Booger
 {
 int a;
 };

 The .d will have both the incorrect empty struct and the correct one
 with the "int a;" declaration.
 extern (C):

 alias _Booger Booger;

 struct _Booger
 {
 }

 struct _Booger
 {
 int a;
 }



This is a known issue, it's on the todo list. I've reported it to github 
as well: https://github.com/jacob-carlborg/dstep/issues/5

-- 
/Jacob Carlborg
Jul 10 2012
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-07-08 23:22, Jonathan Andrew wrote:


 The only disadvantage to the single-file limitation is that in the case
 of GTK at least, it has preprocessor directives to keep you from just
 #include-ing the single file you want to convert, so I just used sed to
 strip out all the #error directives that come up and force it to do my
 bidding. I understand DStep doesn't deal with preprocessor yet, but as
 far as the CLang front-end it uses goes, it might be helpful to find a
 way to turn off #error-s.

 sed -i 's/#error/\/\//g' *.h



I added an enhancement request for this:

https://github.com/jacob-carlborg/dstep/issues/3


 The next step was to rename all the D reserved words that GTK used as
 function arguments - in, out, function, and align are the only ones I
 can think of off the top of my head. Easy fix for the user (by no means
 am I complaining), but if you want to streamline the conversion,
 automatically renaming these kinds of arguments might be a helpful option.



I added this to the issue list as well:

https://github.com/jacob-carlborg/dstep/issues/1


 Then, renaming all the duplicate empty struct{} entries in some of the
 files. You already know about this, but it was probably the most
 time-consuming part of the process for converting GTK, at least. I
 couldn't think of an easy way to automate this on my end, because some
 of the empty structs were necessary to get it to compile.



BTW, do you have an example of the C code that causes these empty structs?


 Finally, putting import statements in all the .d files after I was done.
 Still a long way to go on this (500 files).



Imports are already added for most of the files in the C standard 
library. I plan to add support for other files as well. Hopefully this 
can be done.


 Sorry for the long post, this is probably obvious stuff to everybody
 else, but I was really impressed with DStep - thank you for creating it!

 -Jon




-- 
/Jacob Carlborg
Jul 09 2012
prev sibling next sibling parent reply "Andrea Fontana" <nospam example.com> writes:
Ok I know it's v 0.0.1, but I think this bugs are not so 
difficult to fix:

- d keywords should be escaped  => (for example  int f(int out) 
should become int f(int _out) or something similar...)
- self alias should be removed  => typedef test { int a; } test; 
generate alias test test; struct test { int a; };
- struct gives error if used on function ( "xxxx is used as a 
type")
- variadic function gives errors: "Error: variadic functions with 
non-D linkage must have at least one parameter"

:)

On Saturday, 7 July 2012 at 14:47:49 UTC, Jacob Carlborg wrote:

 DStep is a tool for translating C and Objective-C headers to D 
 modules. It uses libclang for lexing/parsing and AST traversal. 
 This means it handles everything that Clang itself can handle, 
 although this doesn't mean it will correctly translate 
 everything.

 I would consider this release alpha or beta. I'm releasing this 
 now in hope I get some feedback on what language features the 
 tool can't handle.

 The tool is available at github:
 https://github.com/jacob-carlborg/dstep

 Binaries are available for Mac OS X and Ubuntu 11.10 32bit:
 https://github.com/jacob-carlborg/dstep/downloads

 Unfortunately I haven't been able to successfully compile it on 
 Windows due to Optlink not cooperating. I'll most likely 
 provide Linux binaries with better compatibility later.

 Build instructions are available at github.

 Usage:

 dstep <input-file.h> -o output_file.d

 For Objective-C

 dstep <input-file.h> -o output_file.d -ObjC

 Tests:

 DStep uses Cucumber and Aruba (Ruby tools) to run its tests. It 
 will basically run the tool on all *.h files in the 
 "test_files" directory and compare the results to the 
 corresponding *.d files.

 Known issues/missing functionality:

 * Multiple input files
 * Framework as input file
 * Add module declaration
 * Option for specifying before and after code
 * Option for specifying package
 * Windows support

 C:
     * Self includes
     * Out of order typedefs of structs
     * Bitfields
     * Non-standard extensions
     * Preprocessor
     * Arrays with no size marked as "extern".

 Objective-C:
     * Protocols
     * Properties
     * Blocks
     * Categories
     * Actions
     * Outlets
     * Selectors

 This is basically what's on the todo list:

 https://raw.github.com/jacob-carlborg/dstep/master/todo.taskpaper

 There's no point in reporting issues which are listed above.
Jul 09 2012
parent reply Jacob Carlborg <doob me.com> writes:
On 2012-07-09 17:49, Andrea Fontana wrote:

 Ok I know it's v 0.0.1, but I think this bugs are not so difficult to fix:

 - d keywords should be escaped  => (for example  int f(int out) should
 become int f(int _out) or something similar...)
 - self alias should be removed  => typedef test { int a; } test;
 generate alias test test; struct test { int a; };



I've added this to issues to github: 
https://github.com/jacob-carlborg/dstep/issues


 - struct gives error if used on function ( "xxxx is used as a type")



I'm not sure I understand what you mean. Do you have an example of the C 
code and the generated D code.


 - variadic function gives errors: "Error: variadic functions with non-D
 linkage must have at least one parameter"



Do you have an example of the C code that causes this error? DStep 
should give an error for C code like this:

void foo (...);

File(117B84)foo.h:1:11: error: ISO C requires a named argument before '...'

-- 
/Jacob Carlborg
Jul 09 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-07-10 09:57, Andrea Fontana wrote:


 struct test
 {
 int var;
 };

 struct test* first();

 Dstep translate first() => first(...) in this case.



As far as I know that is legal C code is a variadic function. It's the 
old K&R style which is discouraged now but still legal:

http://en.wikipedia.org/wiki/ANSI_C#Compliance_detectability

If you want a function taking no arguments the correct syntax is:

struct test* first (void);

I don't know the best way to deal with that. It's legal C code but not 
legal D. I could check if there is no parameters and assume that's what 
the users wants but it's not correct. Perhaps I could provide a flag for 
this.

-- 
/Jacob Carlborg
Jul 10 2012
prev sibling next sibling parent "Andrea Fontana" <nospam example.com> writes:
On Monday, 9 July 2012 at 19:24:52 UTC, Jacob Carlborg wrote:

 On 2012-07-09 17:49, Andrea Fontana wrote:

 - struct gives error if used on function ( "xxxx is used as a 
 type")



 I'm not sure I understand what you mean. Do you have an example 
 of the C code and the generated D code.



Ok that was a problem with my struct name conflicting with module 
name. Sorry!


 - variadic function gives errors: "Error: variadic functions 
 with non-D
 linkage must have at least one parameter"



 Do you have an example of the C code that causes this error? 
 DStep should give an error for C code like this:

 void foo (...);



struct test
{
         int var;
};

struct test* first();

Dstep translate first() =>  first(...) in this case.
Jul 10 2012
prev sibling next sibling parent "Andrea Fontana" <nospam example.com> writes:
On Monday, 9 July 2012 at 19:24:52 UTC, Jacob Carlborg wrote:

 On 2012-07-09 17:49, Andrea Fontana wrote:

 - struct gives error if used on function ( "xxxx is used as a 
 type")



 I'm not sure I understand what you mean. Do you have an example 
 of the C code and the generated D code.



Ok that was a problem with my struct name conflicting with module 
name. Sorry!


 - variadic function gives errors: "Error: variadic functions 
 with non-D
 linkage must have at least one parameter"



 Do you have an example of the C code that causes this error? 
 DStep should give an error for C code like this:

 void foo (...);



struct test
{
         int var;
};

struct test* first();

Dstep translate first() =>  first(...) in this case.
Jul 10 2012
prev sibling next sibling parent "Jonathan Andrew" <jonfandrew gmail.com> writes:
On Monday, 9 July 2012 at 06:30:39 UTC, Jacob Carlborg wrote:

 On 2012-07-08 23:22, Jonathan Andrew wrote:


 Jacob,




 The only disadvantage to the single-file limitation is that in 
 the case
 of GTK at least, it has preprocessor directives to keep you 
 from just
 #include-ing the single file you want to convert, so I just 
 used sed to
 strip out all the #error directives that come up and force it 
 to do my
 bidding. I understand DStep doesn't deal with preprocessor 
 yet, but as
 far as the CLang front-end it uses goes, it might be helpful 
 to find a
 way to turn off #error-s.



 I had no idea about that.


 sed -i 's/#error/\/\//g' *.h

 The next step was to rename all the D reserved words that GTK 
 used as
 function arguments - in, out, function, and align are the only 
 ones I
 can think of off the top of my head. Easy fix for the user (by 
 no means
 am I complaining), but if you want to streamline the 
 conversion,
 automatically renaming these kinds of arguments might be a 
 helpful option.



 I thought the tool did that already.


 Then, renaming all the duplicate empty struct{} entries in 
 some of the
 files. You already know about this, but it was probably the 
 most
 time-consuming part of the process for converting GTK, at 
 least. I
 couldn't think of an easy way to automate this on my end, 
 because some
 of the empty structs were necessary to get it to compile.



 I thought I had fixed this too. I'll have to take a look.


 Finally, putting import statements in all the .d files after I 
 was done.
 Still a long way to go on this (500 files).

 Sorry for the long post, this is probably obvious stuff to 
 everybody
 else, but I was really impressed with DStep - thank you for 
 creating it!



 No it's good, this is just what I wanted people to do. It would 
 be great if you could report these issues:

 https://github.com/jacob-carlborg/dstep/issues

 If you have a simple test case or a header I can try that would 
 be great.



OK, as far as the empty struct-s, it looks like it has to do with 
typedef struct.
-------------------------------------------------------
//Test.h:

typedef struct _Booger Booger;

//Results in:

-------------------------------------------------------
//Test.d:

extern (C):

alias _Booger Booger;

struct _Booger
{
}

-------------------------------------------------------

If the .h has:

typedef struct _Booger Booger;

struct Booger
{
    int a;
};

The .d will have both the incorrect empty struct and the correct 
one with the "int a;" declaration.
extern (C):

alias _Booger Booger;

struct _Booger
{
}

struct _Booger
{
    int a;
}

Thanks,
    Jon
Jul 10 2012
prev sibling next sibling parent "Jonathan Andrew" <jonfandrew gmail.com> writes:
 -------------------------------------------------------

 If the .h has:

 typedef struct _Booger Booger;

 struct Booger
 {
    int a;
 };



Oops, typo! should be:

  typedef struct _Booger Booger;

  struct _Booger
  {
     int a;
  };
Jul 10 2012
prev sibling next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/7/12, Jacob Carlborg <doob me.com> wrote:

 snip



Nice work!

Can I ask you something? Do you know if (lib)clang exports typeinfo
for default values? For example:

namespace Foo
{
    enum En
    {
        Val1,
        Val2
    };
}

void test(int x = Foo::Val1) { }

'x' has typeinfo (it's an int), but I'm interested in the typeinfo for
the default value "Foo::Val1". In other tools (like gccxml) the
default value is exported as a string which makes generating default
values in D hard (not impossible but just hard).
Jul 20 2012
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-07-20 15:04, Andrej Mitrovic wrote:


 Nice work!



Thanks.


 Can I ask you something? Do you know if (lib)clang exports typeinfo
 for default values? For example:

 namespace Foo
 {
      enum En
      {
          Val1,
          Val2
      };
 }

 void test(int x = Foo::Val1) { }

 'x' has typeinfo (it's an int), but I'm interested in the typeinfo for
 the default value "Foo::Val1". In other tools (like gccxml) the
 default value is exported as a string which makes generating default
 values in D hard (not impossible but just hard).



Don't know, sorry. I have only worked with C and Objective-C code. I can 
see if I can find out.

-- 
/Jacob Carlborg
Jul 20 2012
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-07-20 15:04, Andrej Mitrovic wrote:

 On 7/7/12, Jacob Carlborg <doob me.com> wrote:

 snip



 Nice work!

 Can I ask you something? Do you know if (lib)clang exports typeinfo
 for default values? For example:

 namespace Foo
 {
      enum En
      {
          Val1,
          Val2
      };
 }

 void test(int x = Foo::Val1) { }

 'x' has typeinfo (it's an int), but I'm interested in the typeinfo for
 the default value "Foo::Val1". In other tools (like gccxml) the
 default value is exported as a string which makes generating default
 values in D hard (not impossible but just hard).



In libclang, the kind of "Foo::Val1" in the above example is:

CXCursor_FirstExpr -> CXCursor_DeclRefExpr -> CXCursor_NamespaceRef.

What I did here was I checked the kind of "Foo::Val1" cursor, drilled 
down into the cursor children as far as possible.

Cursor kinds:

http://clang.llvm.org/doxygen/group__CINDEX.html#gaaccc432245b4cd9f2d470913f9ef0013

-- 
/Jacob Carlborg
Jul 20 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-07-20 16:58, Andrej Mitrovic wrote:


 Cool, this will come in handy. Thanks.



If your interested you can help out with DStep, add C++ support or 
similar :)

-- 
/Jacob Carlborg
Jul 22 2012
prev sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/20/12, Jacob Carlborg <doob me.com> wrote:

 In libclang, the kind of "Foo::Val1" in the above example is:

 CXCursor_FirstExpr -> CXCursor_DeclRefExpr -> CXCursor_NamespaceRef.

 What I did here was I checked the kind of "Foo::Val1" cursor, drilled
 down into the cursor children as far as possible.

 Cursor kinds:

 http://clang.llvm.org/doxygen/group__CINDEX.html#gaaccc432245b4cd9f2d470913f9ef0013



Cool, this will come in handy. Thanks.
Jul 20 2012