www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - (Oh My) Gentool 0.3.0 released

reply evilrat <evilrat666 gmail.com> writes:
(Oh My) Gentool - Yet another C/C++ binding generator.

It is a tool to convert C/C++ code to D usable form. It takes 
JSON config, basically all C++ compiler flags and switches, and 
outputs extern(C++) declarations, (hopefully) in usable form D!

It can already process (dear) imgui library (immediate mode GUI 
popular in game development and various graphics related tools 
and demos) without manual fixes!

It can process recastnavigation (navmesh generation and 
pathfinding library) with just a few manual edits.

Please note that it is still in its early stage and may contain 
bugs and missing language features, as well as lack of conversion 
for certain language constructs.

It is still hard to use it directly in the build process on a 
real libraries due to many syntax and semantics issues, however 
it is already a valuable tool for making thin wrappers on C++ 
side to quickly bring them to your D code, given that your 
wrapper headers does not contains complex bodies or templates, or 
direct inclusions of other libraries headers such as Boost(ok, no 
STL too).

That's it, even if it produces incomplete translation this could 
reduce bindings making process from hours down to minutes! Who 
wants to spent 10 hours manually making bindings for entire PhysX 
when it can be reduced to just 30 minutes? Absolutely no one! 
Grab one today and stay ahead of your competitors with regular 
updates!


How to start
https://github.com/Superbelko/ohmygentool/wiki/QuickStart

Source
https://github.com/Superbelko/ohmygentool

Windows build
https://github.com/Superbelko/ohmygentool/releases/tag/v0.3.0
May 04
parent reply user1234 <user1234 12.de> writes:
On Wednesday, 5 May 2021 at 06:50:29 UTC, evilrat wrote:
 (Oh My) Gentool - Yet another C/C++ binding generator.

 It is a tool to convert C/C++ code to D usable form. It takes 
 JSON config, basically all C++ compiler flags and switches, and 
 outputs extern(C++) declarations, (hopefully) in usable form D!

 ...

 Source
 https://github.com/Superbelko/ohmygentool
 ...
I have a technical question about the tool itself. It is mostly written in cpp. Is it possible to use libclang and more generally LLVM c++ api [directly in D](https://dlang.org/spec/cpp_interface.html) or the Cpp interface is too limited ? Was this an option, have you tried ?
May 05
next sibling parent reply Dominikus Dittes Scherkl <dominikus scherkl.de> writes:
On Wednesday, 5 May 2021 at 10:01:13 UTC, user1234 wrote:
 I have a technical question about the tool itself. It is mostly 
 written in cpp.
Oh dear! Isn't it possible to use it to translate itself into D?
May 05
parent reply evilrat <evilrat666 gmail.com> writes:
On Wednesday, 5 May 2021 at 10:35:23 UTC, Dominikus Dittes 
Scherkl wrote:
 On Wednesday, 5 May 2021 at 10:01:13 UTC, user1234 wrote:
 I have a technical question about the tool itself. It is 
 mostly written in cpp.
Oh dear! Isn't it possible to use it to translate itself into D?
To answer both: clang has lots of templates, sometimes not so trivial ones, its code base filled with C++ constructs that does not have nice one to one translation(or simply a C++ specific detail like alignment), and the most annoying part - it has various 'tables' generated as a build step that is a mix of external tools and macros. Basically every template will need some care, likely it will be easier to just emit C++ stubs that will force compiler to emit actual code to link with than trying to translate them. So no, not currently possible. Definitely not an unpaid job, well I'll still reject it even if it is paid one. Maybe some time in future, but for now there is a lot more priority stuff to do before even attempting this.
May 05
parent reply user1234 <user1234 12.de> writes:
On Wednesday, 5 May 2021 at 11:51:27 UTC, evilrat wrote:
 On Wednesday, 5 May 2021 at 10:35:23 UTC, Dominikus Dittes 
 Scherkl wrote:
 On Wednesday, 5 May 2021 at 10:01:13 UTC, user1234 wrote:
... To answer both: clang has lots of templates, sometimes not so trivial ones, its code base filled with C++ constructs that does not have nice one to one translation(or simply a C++ specific detail like alignment), and the most annoying part - it has various 'tables' generated as a build step that is a mix of external tools and macros. Basically every template will need some care, likely it will be easier to just emit C++ stubs that will force compiler to emit actual code to link with than trying to translate them. So no, not currently possible. Definitely not an unpaid job, well I'll still reject it even if it is paid one. Maybe some time in future, but for now there is a lot more priority stuff to do before even attempting this.
Thanks for the explanations. BTW I had the same question for LDC backend being c++, I guess the answer would be similar.
May 05
next sibling parent Dominikus Dittes Scherkl <dominikus scherkl.de> writes:
On Wednesday, 5 May 2021 at 11:54:51 UTC, user1234 wrote:
 On Wednesday, 5 May 2021 at 10:01:13 UTC, user1234 wrote:
 Maybe some time in future, but for now there is a lot more 
 priority stuff to do before even attempting this.
Thanks for the explanations. BTW I had the same question for LDC backend being c++, I guess the answer would be similar.
Yeah, but unlike LDC, for a tool that translates C++ to D it is a goal being able to also translate itself from C++ to D. And reaching this goal would be a huge milestone!
May 06
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2021-05-05 13:54, user1234 wrote:

 Thanks for the explanations. BTW I had the same question for LDC backend 
 being c++, I guess the answer would be similar.
If I understand correctly, the Zig compiler is implemented partially in Zig. It use the LLVM C API and some wrappers C around the C++ API where the C API is not sufficient. -- /Jacob Carlborg
May 07
parent evilrat <evilrat666 gmail.com> writes:
On Friday, 7 May 2021 at 18:17:36 UTC, Jacob Carlborg wrote:
 On 2021-05-05 13:54, user1234 wrote:

 Thanks for the explanations. BTW I had the same question for 
 LDC backend being c++, I guess the answer would be similar.
If I understand correctly, the Zig compiler is implemented partially in Zig. It use the LLVM C API and some wrappers C around the C++ API where the C API is not sufficient.
And it quickly becomes insufficient using only the C API as feature complexity increases. No idea if Zig has to deal with C++ compiler (clang) or all it needs is pure LLVM, because the latter should have more or less feature rich C API, unlike clang that deals with C++ and has ever "unstable" API. Making wrappers for missing parts still will be a PITA, as having to pass around smart pointers definitely does not makes it easier. Anyway like I said, for bootstrap goal it is probably easier to re-purpose the tool to make thin wrappers & stubs on C++ side, pretty much just like SWIG does. Side note: But all this does not compares to what potential D to nextgen-language bindings making process would look like, as D feature set makes it even harder to translate, esp. stuff like templates and CTFE, now add static if's to that and it becomes a real mess.
May 07
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2021-05-05 12:01, user1234 wrote:

 Is it possible to use libclang and more generally LLVM c++ api [directly 
 in D](https://dlang.org/spec/cpp_interface.html) or the Cpp interface is 
 too limited ?
 Was this an option, have you tried ?
Yes, it's possible to use libclang. DStep [1] is using that and it fully written in D. Although DStep cannot create bindings for C++ yet so I cannot guarantee that using only libclang will work for C++ code. [1] https://github.com/jacob-carlborg/dstep -- /Jacob Carlborg
May 07
parent reply Gavin Ray <user example.com> writes:
On Friday, 7 May 2021 at 18:15:47 UTC, Jacob Carlborg wrote:
 On 2021-05-05 12:01, user1234 wrote:

 Is it possible to use libclang and more generally LLVM c++ api 
 [directly in D](https://dlang.org/spec/cpp_interface.html) or 
 the Cpp interface is too limited ?
 Was this an option, have you tried ?
Yes, it's possible to use libclang. DStep [1] is using that and it fully written in D. Although DStep cannot create bindings for C++ yet so I cannot guarantee that using only libclang will work for C++ code. [1] https://github.com/jacob-carlborg/dstep
I don't think using libclang is a good idea for C++. I think it works very well and is the ideal choice (due to lack of dependencies and infra) for C, but C++ has a nightmarish AST. Have been investigating doing this with LibTooling using the C++ API, however a bit differently than has been done before (will explain in a minute below). The reasoning is that, when I spoke with LLVM developers, and developers who had built successful codegen tooling for C++, every one of them cautioned against using libclang, said that they regretted the choice, and that the API and AST info it exposes is not sufficient.
 _I had a chance to ask Atila this same thing, last Beerconf, 
 and that was also his stance -- roughly that a large part of 
 the hurdles with dpp (beyond C++ being a nightmare to start) 
 were that it was built on libclang._
--- I have a large personal interest in building/helping to build a viable C++-header-to-D-`extern (C++)` generator because I think that is what would unlock a vast amount of potential + power for D. Quick, no-hassle direct bindings to any (or even most) C++ libraries and tools. But I've been doing my homework both on how to approach this and what the state of C++ bindgen is in D. I've tried: - D++ - Dstep - Ohmygentool - SWIG, with and without "Directors" feature enabled - CPP2D Of those, I've had the best success C++ using Ohmygentool by a fairly large margin. It can often do several-thousand line projects with non-trivial `#includes` mostly automatically. --- **However, I had an idea which I haven't seen tried yet, and have been prototyping:** - Using `cppyy` in Python (which uses `cling`) for runtime bindings to C++ and ability to write raw C++ code in Python strings and JIT compile it. - Allow users to write "drivers"/"clients" in Python which do the codegen. Since Python isn't compiled, this means you can realtime tweak and visualize your output much faster than manually recompiling a C++ based LibTooling application. I am thinking of some kind of API where you can declare rules using annotations for AST nodes above functions for handling them. Something like: ```py class DCodegen: Clang/LibTooling's AST API rule(lambda t: t.is_pointer() or t.is_reference() and \ t.pointee().is_record_indirection()) def input(cls, t, args): return f"{{interm}} = &{c_util.struct_cast(t, '{inp}')};" rule(lambda t: t.is_pointer() or t.is_reference()) def input(cls, t, args): raise ValueError("unsupported input pointer/reference type {}".format(t)) ``` This would allow people to contribute or tweak the codegen to their liking very rapidly. For distribution, it could be done in an Ubuntu Docker container that comes with LLVM and Python in it, and the scripts, then mapped to local filesystem for read access + also if you want to edit the `DCodegen` script or supply your own Python file as the driver. --- **What do you all think of this idea?** - Would love to hear feedback/opinions. - I unfortunately have neither the D nor C++ expertise to properly write the translation rules, so it would take some help/collaboration from the community. --- Here's an example of loading LibTooling in Python and building an AST from some code. You can see the C++ object in the `print()` output in the top right: - ![](https://i.imgur.com/FkSq0N8.png) - ![](https://i.imgur.com/7rNbegn.png)
May 09
parent reply evilrat <evilrat666 gmail.com> writes:
On Sunday, 9 May 2021 at 19:35:52 UTC, Gavin Ray wrote:
 **However, I had an idea which I haven't seen tried yet, and 
 have been prototyping:**
 - Using `cppyy` in Python (which uses `cling`) for runtime 
 bindings to C++ and ability to write raw C++ code in Python 
 strings and JIT compile it.

 - Allow users to write "drivers"/"clients" in Python which do 
 the codegen. Since Python isn't compiled, this means you can 
 realtime tweak and visualize your output much faster than 
 manually recompiling a C++ based LibTooling application.

 I am thinking of some kind of API where you can declare rules 
 using annotations for AST nodes above functions for handling 
 them. Something like:
 ```py
 class DCodegen:

 Clang/LibTooling's AST API
      rule(lambda t: t.is_pointer() or t.is_reference() and \
                     t.pointee().is_record_indirection())
     def input(cls, t, args):
         return f"{{interm}} = &{c_util.struct_cast(t, 
 '{inp}')};"

      rule(lambda t: t.is_pointer() or t.is_reference())
     def input(cls, t, args):
         raise ValueError("unsupported input pointer/reference 
 type {}".format(t))
 ```

 This would allow people to contribute or tweak the codegen to 
 their liking very rapidly.
It's all libclang under the hood though. Never looked at the sources of cppyy or cling, but very likely this works for them because of libs such as pybind11 that wraps C++ stuff in C++, while the other tools trying to rely on C API(very limited) or keep fighting with C++ API, you've already seen that all that tools from that list except gentool is using C API and what capabilities each provides. Can't say I hate that idea, but it has same issues as SWIG, writing any non trivial rule becomes next to impossible as there is practically zero examples and very poor documentation, the whole process becomes trial and error marathon without chance to win, and it is basically write-only code that is as worse as C++ templates. But it is definitely better than SWIG in that regard as you can get type and functions information using dir() and help() and your trusty IDE with debugger. My current plan though is to provide predefined pre-generate and post-generate rules that is applied declaratively in project config, for example `ignoreDecls *::new[]` that will ignore all new operator overloads in any namespace, or one of any other existing rules that deals with specific patterns, and at some point later allow users to write their own rules like you described. After all this is binding/translator tool, not an universal one-for-all code generator.
 For distribution, it could be done in an Ubuntu Docker 
 container that comes with LLVM and Python in it, and the 
 scripts, then mapped to local filesystem for read access + also 
 if you want to edit the `DCodegen` script or supply your own 
 Python file as the driver.
No way, Docker is too heavy and not very user friendly or even CI friendly. It is ok to have builds optionally packed in container, but not as the only way to distribute. It might work for cppyy/cling because they rely on specific dynamic library properties or fork process on *NIX that is not there on Windows.
May 10
parent Gavin Ray <user example.com> writes:
On Monday, 10 May 2021 at 07:11:26 UTC, evilrat wrote:
 Can't say I hate that idea, but it has same issues as SWIG, 
 writing any non trivial rule becomes next to impossible as 
 there is practically zero examples and very poor documentation, 
 the whole process becomes trial and error marathon without 
 chance to win, and it is basically write-only code that is as 
 worse as C++ templates. But it is definitely better than SWIG 
 in that regard as you can get type and functions information 
 using dir() and help() and your trusty IDE with debugger.
Yeah, this is true. I actually ran into this issue already -- you have extreme devspeed + flexibility with Python but you're essentially "blind" as to the API and it's a lot of trial-and-error if it's not a very trivial C++ library. I think the end goal of being able to provide some way to easily hook into the codegen pipeline so that users can make tweaks easily & contribute them back, or customize it for personal use is probably more important than the "how". This was just the best "how" I could come up with at the time, hah.
 My current plan though is to provide predefined pre-generate 
 and post-generate rules that is applied declaratively in 
 project config,
 for example
   `ignoreDecls *::new[]`
 that will ignore all new operator overloads in any namespace, 
 or one of any other existing rules that deals with specific 
 patterns, and at some point later allow users to write their 
 own rules like you described.
 After all this is binding/translator tool, not an universal 
 one-for-all code generator.
What about adding `cling` for interpreted JIT C++ without handicaps? https://blog.llvm.org/posts/2021-03-25-cling-beyond-just-interpreting-cpp/ This would let both you and users rapidly develop/iterate-on the codegen part without needing to recompile the entire solution. The main application hosts the `cling` process, and you don't do anything except bring the codegen bits in as JIT-interpreted files. It's not like performance is critical here, you know?😅 The end result is that you (or anyone else) can modify the actual codegen code, and re-run the compiled binary without needing to do anything else. `cling` will take care of acting as the engine to sort out `#include`'s, template instatiations, global context, etc. I dunno, I don't really know that much about this area at all. So there's also a good chance this idea couldn't/wouldn't work, and is not a good one -- but in my naive head it sounds good!
 No way, Docker is too heavy and not very user friendly or even 
 CI friendly. It is ok to have builds optionally packed in 
 container, but not as the only way to distribute. It might work 
 for cppyy/cling because they rely on specific dynamic library 
 properties or fork process on *NIX that is not there on Windows.
Yeah, so the issue there was that `cppyy` needs dynamic libs to be able to work. LibTooling hilariously enough can't even be compiled as a dynamic lib on Windows, since there's a limit of ~65,000 exported visible symbols and `LINK.exe` will error. So between wanting the user to have a fairly recent, self-compiled version of LLVM, and not working on Windows at all, the easiest thing seemed to be to ask "Hey, just do `docker run` and that'll run an Ubuntu image with LLVM 13 built with LLVM_ENABLE_SHARED_LIBS and Python 3 and all of this set up for you." Not ideal but also I think maybe the only way to make the `cppyy` thing work haha.
May 10