www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - built in import c header files

reply 12345swordy <alexanderheistermann gmail.com> writes:
I remember reading a reddit comment saying that if d were to 
import c header files directly without any 3rd party libraries it 
would be a game changer. The question is: how difficult to 
implement a C AST parser for the dmd front end?


-Alex
Oct 06 2020
next sibling parent reply Meta <jared771 gmail.com> writes:
On Tuesday, 6 October 2020 at 20:59:54 UTC, 12345swordy wrote:
 I remember reading a reddit comment saying that if d were to 
 import c header files directly without any 3rd party libraries 
 it would be a game changer. The question is: how difficult to 
 implement a C AST parser for the dmd front end?


 -Alex
You're in luck: https://github.com/atilaneves/dpp
Oct 06 2020
parent reply 12345swordy <alexanderheistermann gmail.com> writes:
On Tuesday, 6 October 2020 at 21:04:20 UTC, Meta wrote:
 On Tuesday, 6 October 2020 at 20:59:54 UTC, 12345swordy wrote:
 I remember reading a reddit comment saying that if d were to 
 import c header files directly without any 3rd party libraries 
 it would be a game changer. The question is: how difficult to 
 implement a C AST parser for the dmd front end?


 -Alex
You're in luck: https://github.com/atilaneves/dpp
What part of "built in and not part of a 3rd party library" do you not understand?
Oct 06 2020
parent reply norm <normical gmail.com> writes:
On Tuesday, 6 October 2020 at 22:42:11 UTC, 12345swordy wrote:
 On Tuesday, 6 October 2020 at 21:04:20 UTC, Meta wrote:
 On Tuesday, 6 October 2020 at 20:59:54 UTC, 12345swordy wrote:
 I remember reading a reddit comment saying that if d were to 
 import c header files directly without any 3rd party 
 libraries it would be a game changer. The question is: how 
 difficult to implement a C AST parser for the dmd front end?


 -Alex
You're in luck: https://github.com/atilaneves/dpp
What part of "built in and not part of a 3rd party library" do you not understand?
You could easily work it out for yourself by looking at any C parser and the DMD front. But why bother with C, it is trivial to generate C bindings from headers. C++ is where it becomes complicated and where it would be useful.
Oct 06 2020
parent reply Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 7 October 2020 at 01:29:05 UTC, norm wrote:
 On Tuesday, 6 October 2020 at 22:42:11 UTC, 12345swordy wrote:
 On Tuesday, 6 October 2020 at 21:04:20 UTC, Meta wrote:
 On Tuesday, 6 October 2020 at 20:59:54 UTC, 12345swordy wrote:
 I remember reading a reddit comment saying that if d were to 
 import c header files directly without any 3rd party 
 libraries it would be a game changer. The question is: how 
 difficult to implement a C AST parser for the dmd front end?


 -Alex
You're in luck: https://github.com/atilaneves/dpp
What part of "built in and not part of a 3rd party library" do you not understand?
You could easily work it out for yourself by looking at any C parser and the DMD front.
 But why bother with C, it is trivial to generate C bindings 
 from headers.
It really isn't. Can it be easy? Yes. Is it easy in the general case? No. I've tried calling a large C codebase from D before. It was a nightmare. It's very hard to do manual wrapping when you have thousands of headers in hundreds of directories and everything depends on everything.
Oct 07 2020
parent reply norm <normical gmail.com> writes:
On Wednesday, 7 October 2020 at 10:04:57 UTC, Atila Neves wrote:
 On Wednesday, 7 October 2020 at 01:29:05 UTC, norm wrote:
 On Tuesday, 6 October 2020 at 22:42:11 UTC, 12345swordy wrote:
 On Tuesday, 6 October 2020 at 21:04:20 UTC, Meta wrote:
 On Tuesday, 6 October 2020 at 20:59:54 UTC, 12345swordy 
 wrote:
 I remember reading a reddit comment saying that if d were 
 to import c header files directly without any 3rd party 
 libraries it would be a game changer. The question is: how 
 difficult to implement a C AST parser for the dmd front end?


 -Alex
You're in luck: https://github.com/atilaneves/dpp
What part of "built in and not part of a 3rd party library" do you not understand?
You could easily work it out for yourself by looking at any C parser and the DMD front.
 But why bother with C, it is trivial to generate C bindings 
 from headers.
Really? I have not had any issues before automatically creating C bindings for D using codegen. Sure, occasionally a macro will bite but it isn't that often nor difficult to fix. They were not the kernel but they were not small code bases without macros either. TBH though now I just use dpp so I don't see the value of bolting on a "C" parser to the D front end. C++ I have struggled with that and it requires an annoying amount of manual editing. DPP works well but even it falls over for some C++ code bases I've tried.
Oct 07 2020
parent reply Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 7 October 2020 at 11:14:02 UTC, norm wrote:
 On Wednesday, 7 October 2020 at 10:04:57 UTC, Atila Neves wrote:
 On Wednesday, 7 October 2020 at 01:29:05 UTC, norm wrote:
 On Tuesday, 6 October 2020 at 22:42:11 UTC, 12345swordy wrote:
 On Tuesday, 6 October 2020 at 21:04:20 UTC, Meta wrote:
 On Tuesday, 6 October 2020 at 20:59:54 UTC, 12345swordy 
 wrote:
 I remember reading a reddit comment saying that if d were 
 to import c header files directly without any 3rd party 
 libraries it would be a game changer. The question is: how 
 difficult to implement a C AST parser for the dmd front 
 end?


 -Alex
You're in luck: https://github.com/atilaneves/dpp
What part of "built in and not part of a 3rd party library" do you not understand?
You could easily work it out for yourself by looking at any C parser and the DMD front.
 But why bother with C, it is trivial to generate C bindings 
 from headers.
Really? I have not had any issues before automatically creating C bindings for D using codegen. Sure, occasionally a macro will bite but it isn't that often nor difficult to fix. They were not the kernel but they were not small code bases without macros either.
Imagine this and extrapolate to hundreds of headers: --- // hdr0.h struct struct0 { struct1* s1; }; --- --- // hdr1.h #define SILLY_MACRO struct2; struct struct1 { SILLY_MACRO* s2; } ---- --- // hdr2.h struct struct2 { struct3* s3; } ---
 TBH though now I just use dpp so I don't see the value of 
 bolting on a "C" parser to the D front end. C++ I have 
 struggled with that and it requires an annoying amount of 
 manual editing. DPP works well but even it falls over for some 
 C++ code bases I've tried.
dpp doesn't officially support C++ yet, because... it doesn't work yet. There are several reasons for this. The first, and most annoying, is that libclang isn't all it's cracked up to be. There's a *lot* of functionality missing for C++ that the compiler frontend itself knows about but doesn't expose via the C API. The second, and maybe unfixable reason, is how to translate C++-only concepts to D. Consider std::is_reference_v. That can't be written in D, not even by hand. What if that gets used in SFINAE? What about expression templates given how operator overloading in D is a lot different? I still don't know how to get around that.
Oct 08 2020
parent Walter Bright <newshound2 digitalmars.com> writes:
There's another huge problem with C files.

It's the predefined macros. Every compiler generates a different set of them, 
and in each compiler the set varies by compiler version, target, switch 
settings, etc.

Most are undocumented.

I think gcc has somewhere near a thousand of them.

Naturally, every non-trivial C program winds up totally dependent on them.
Apr 17
prev sibling next sibling parent reply Max Haughton <maxhaton gmail.com> writes:
On Tuesday, 6 October 2020 at 20:59:54 UTC, 12345swordy wrote:
 I remember reading a reddit comment saying that if d were to 
 import c header files directly without any 3rd party libraries 
 it would be a game changer. The question is: how difficult to 
 implement a C AST parser for the dmd front end?


 -Alex
Not impossible (it's worth saying that you can literally do this as a library via mixins), but having a c parser in the compiler seems like too much work when it's already fairly trivial to do. If D had a team of full time staff on payroll, then it could be made to work but it adds so much surface area to the compiler.
Oct 06 2020
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/6/2020 7:56 PM, Max Haughton wrote:
 Not impossible (it's worth saying that you can literally do this as a library 
 via mixins), but having a c parser in the compiler seems like too much work
when 
 it's already fairly trivial to do. If D had a team of full time staff on 
 payroll, then it could be made to work but it adds so much surface area to the 
 compiler.
You'll also have problems with all the nutburger C extensions implemented by various compilers.
Apr 17
parent reply 12345swordy <alexanderheistermann gmail.com> writes:
On Saturday, 17 April 2021 at 21:32:24 UTC, Walter Bright wrote:
 On 10/6/2020 7:56 PM, Max Haughton wrote:
 Not impossible (it's worth saying that you can literally do 
 this as a library via mixins), but having a c parser in the 
 compiler seems like too much work when it's already fairly 
 trivial to do. If D had a team of full time staff on payroll, 
 then it could be made to work but it adds so much surface area 
 to the compiler.
You'll also have problems with all the nutburger C extensions implemented by various compilers.
There is no need for the dmd frontend to implement that. The ldc2 and the gdc compilers can relay on their c++ counterparts to provided the c extensions. -Alex
Apr 17
parent reply Max Haughton <maxhaton gmail.com> writes:
On Saturday, 17 April 2021 at 22:12:41 UTC, 12345swordy wrote:
 On Saturday, 17 April 2021 at 21:32:24 UTC, Walter Bright wrote:
 On 10/6/2020 7:56 PM, Max Haughton wrote:
 Not impossible (it's worth saying that you can literally do 
 this as a library via mixins), but having a c parser in the 
 compiler seems like too much work when it's already fairly 
 trivial to do. If D had a team of full time staff on payroll, 
 then it could be made to work but it adds so much surface 
 area to the compiler.
You'll also have problems with all the nutburger C extensions implemented by various compilers.
There is no need for the dmd frontend to implement that. The ldc2 and the gdc compilers can relay on their c++ counterparts to provided the c extensions. -Alex
No they can't or at least GCC *maybe* could but LDC is a separate project entirely from clang.
Apr 17
parent 12345swordy <alexanderheistermann gmail.com> writes:
On Saturday, 17 April 2021 at 23:22:47 UTC, Max Haughton wrote:
 On Saturday, 17 April 2021 at 22:12:41 UTC, 12345swordy wrote:
 On Saturday, 17 April 2021 at 21:32:24 UTC, Walter Bright 
 wrote:
 On 10/6/2020 7:56 PM, Max Haughton wrote:
 Not impossible (it's worth saying that you can literally do 
 this as a library via mixins), but having a c parser in the 
 compiler seems like too much work when it's already fairly 
 trivial to do. If D had a team of full time staff on 
 payroll, then it could be made to work but it adds so much 
 surface area to the compiler.
You'll also have problems with all the nutburger C extensions implemented by various compilers.
There is no need for the dmd frontend to implement that. The ldc2 and the gdc compilers can relay on their c++ counterparts to provided the c extensions. -Alex
No they can't or at least GCC *maybe* could but LDC is a separate project entirely from clang.
Actually they can. It is a question of money/time/resources not programming ability here. Regardless the dmd front-end just need to strictly implement the c99 standard(no language extension by the compiler of any sorts) to get the ball rolling. -Alex
Apr 17
prev sibling next sibling parent reply Alexis Kyaw <alexis kyaw.de> writes:
On Tuesday, 6 October 2020 at 20:59:54 UTC, 12345swordy wrote:
 I remember reading a reddit comment saying that if d were to 
 import c header files directly without any 3rd party libraries 
 it would be a game changer. The question is: how difficult to 
 implement a C AST parser for the dmd front end?


 -Alex
This would indeed be a total game changer. I just looked at D in the course of checking out newer programming languages and it turns out that I think none of them will be really successful replacing C (or C++) because there's no direct compatibility with C (Priority 1) and C++ (Priority 2) and ideally not only using the C (and potentially C++) binaries directly, but also creating header files to allow using the D binaries smoothly "on the other side". There is just some much existing code that you would totally on the winner side with this and your totally out in the desert without it. Just like the biggest argument for Java is it's ecosystem, this would make all C (and maybe even C++) libraries part of the D ecosystem. For this to be really an argument, it must be an in-built feature of the language toolchain and a priority in all future developments. Just imagine what this would mean: a clean way forward from existing C code to write bulletproof applications on top of all the existing code. And a way to use the new code from new libraries written in D without any hassle. A potential external tool would a C to D converter which also allows easy migration of C code. Just imagine converting a complete Linux implementation to D, compile it and having a full D-based Linux. Then you could start migrating the code base to use the advanced features of D and start a new age for the whole industry. This would make the world listen. Actually it looks as if this is not such an ambitious goal for D, so I'm surprised it's not under way. If had time I'd maybe look at it myself, unfortunately I have project work and 3 children, leaving really very little time for extra projects.
Apr 17
next sibling parent Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Saturday, 17 April 2021 at 08:36:41 UTC, Alexis Kyaw wrote:
 On Tuesday, 6 October 2020 at 20:59:54 UTC, 12345swordy wrote:
 I remember reading a reddit comment saying that if d were to 
 import c header files directly without any 3rd party libraries 
 it would be a game changer. The question is: how difficult to 
 implement a C AST parser for the dmd front end?


 -Alex
This would indeed be a total game changer. I just looked at D in the course of checking out newer programming languages and it turns out that I think none of them will be really successful replacing C (or C++) because there's no direct compatibility with C (Priority 1) and C++ (Priority 2) and ideally not only using the C (and potentially C++) binaries directly, but also creating header files to allow using the D binaries smoothly "on the other side". There is just some much existing code that you would totally on the winner side with this and your totally out in the desert without it. Just like the biggest argument for Java is it's ecosystem, this would make all C (and maybe even C++) libraries part of the D ecosystem. For this to be really an argument, it must be an in-built feature of the language toolchain and a priority in all future developments. Just imagine what this would mean: a clean way forward from existing C code to write bulletproof applications on top of all the existing code. And a way to use the new code from new libraries written in D without any hassle. A potential external tool would a C to D converter which also allows easy migration of C code. Just imagine converting a complete Linux implementation to D, compile it and having a full D-based Linux. Then you could start migrating the code base to use the advanced features of D and start a new age for the whole industry. This would make the world listen. Actually it looks as if this is not such an ambitious goal for D, so I'm surprised it's not under way. If had time I'd maybe look at it myself, unfortunately I have project work and 3 children, leaving really very little time for extra projects.
+1 Have been experimenting with converting C(++) to D. It's possible to some extent. *dream hat on* I've thought about this, and what would be best for D is if code could be converted. Why? Because otherwise we still just have built a lot of stuff upon the old, meaning that the % of D in the world hasn't really increased much. Also, any modifications would be hard etc etc. What's lacking tho is good tooling around D, like static analysis etc etc. But, if C(++) could be converted to D, *some* parts of the tools might be reusable, and you would get a compounding effect after a while. For instance, even *if* C(++) and D could be seamlessly mixed in the same source file, you would still end up with much C(++) code. I still think dpp, dstep etc are 100% needed, and more work should definitely be put in it, but the endgame would be to somehow increase the amount of D, right? (Btw, what's the state of cpp2d?) Yes, that would mean you end up with two code bases, but that's kind of the point in this case. There are obviously + and - with this solution. But maybe the endgame deluxe would be to be able to reuse the parts that work well and convert the parts that doesn't. Best of both worlds. *dream hat off* <- note
Apr 17
prev sibling parent Atila Neves <atila.neves gmail.com> writes:
On Saturday, 17 April 2021 at 08:36:41 UTC, Alexis Kyaw wrote:
 On Tuesday, 6 October 2020 at 20:59:54 UTC, 12345swordy wrote:
 I remember reading a reddit comment saying that if d were to 
 import c header files directly without any 3rd party libraries 
 it would be a game changer. The question is: how difficult to 
 implement a C AST parser for the dmd front end?


 -Alex
This would indeed be a total game changer.
Why would one implement a C parser (never mind a C++ one) when libclang exists?
Apr 21
prev sibling parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Tuesday, 6 October 2020 at 20:59:54 UTC, 12345swordy wrote:
 I remember reading a reddit comment saying that if d were to 
 import c header files directly without any 3rd party libraries 
 it would be a game changer. The question is: how difficult to 
 implement a C AST parser for the dmd front end?


 -Alex
I've been doing this in both of my compiled languages. (fcc first, built-in, and now in cx as a macro.) For an example of it in action, check out my glfw sample https://github.com/FeepingCreature/Cx/blob/master/demos/glfw.cx , or the macro code for it https://github.com/FeepingCreature/Cx/blob/master/src/cx/macros/cimport.cx . Three things that jump out: 1. preprocessed headers with `gcc -dD -E` are a lot easier to work with, because gcc helpfully unrolls *most* of #define definitions, but leaves enough standing so that you can generate enums for all the #defines. This is why it's a useful thing to have unrestricted IO support, including arbitrary command execution, in CTFE. :) 2. "Header C" is actually pretty easy to parse if you just studiously ignore all the weird things, weird inside-out type format aside. 3. 10% of effort to parse C gets you 99% of value: because the vast majority of typing work involved in making a C header binding is trivially simple to parse from the header, and the rest you can always just do by hand, or (if your language has macros) expand as you need it. I presume glibc headers are more complicated, but I have been massively positively surprised how much of GLFW and GL I could get running with a day or two of coding work on this. To a first approximation, once I added function pointers, it pretty much all just worked. I can wholeheartedly recommend any language do this. It unlocks a lot of value and removes a lot of manual work.
Apr 18
parent reply evilrat <evilrat666 gmail.com> writes:
On Monday, 19 April 2021 at 06:51:54 UTC, FeepingCreature wrote:
 On Tuesday, 6 October 2020 at 20:59:54 UTC, 12345swordy wrote:
 I remember reading a reddit comment saying that if d were to 
 import c header files directly without any 3rd party libraries 
 it would be a game changer. The question is: how difficult to 
 implement a C AST parser for the dmd front end?


 -Alex
I've been doing this in both of my compiled languages. (fcc first, built-in, and now in cx as a macro.) For an example of it in action, check out my glfw sample https://github.com/FeepingCreature/Cx/blob/master/demos/glfw.cx , or the macro code for it https://github.com/FeepingCreature/Cx/blob/master/src/cx/macros/cimport.cx . ... I can wholeheartedly recommend any language do this. It unlocks a lot of value and removes a lot of manual work.
Yeah, just until it fails to convert something and blocks compilation, where no manual fixes is possible or not even allowed by design.
Apr 19
parent FeepingCreature <feepingcreature gmail.com> writes:
On Monday, 19 April 2021 at 09:09:21 UTC, evilrat wrote:
 On Monday, 19 April 2021 at 06:51:54 UTC, FeepingCreature wrote:
 I can wholeheartedly recommend any language do this. It 
 unlocks a lot of value and removes a lot of manual work.
Yeah, just until it fails to convert something and blocks compilation, where no manual fixes is possible or not even allowed by design.
Yeah I think it differs between using it for system headers and using it for library headers, that are probably going to be relatively standardized across distributions and compiler versions. That said, at least the p much worst thing that can happen is a compiler error. With manual headers, you can end up on a weird system where the layout of a header struct is different, or a function is implemented as a header macro that calls a different function, and then you get linker errors or miscompilation. If anything, the headers are probably closer to the "source of truth" about the API.
Apr 19