www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Header-Only Library - pragma(root)

reply Walter Bright <newshound2 digitalmars.com> writes:
There is much misunderstanding in the "What does 'inline' mean?" thread.

I've slowly come to realize that it's not about inlining at all, it's about a 
desire for "header-only" libraries. To use a header-only library, you only have 
to import the module for it.

This works today in D if you also add the imported module to the command line. 
But, of course, it's good to make it simpler.

So, I propose the addition of `pragma(root);` which looks like:

--- electron.di ---
module electron;
pragma(root);    // header-only library!

int orbit() { return 3; }

--- atom.d ---
module atom;
import core.stdc.stdio;

import electron;

void main() {
     printf("%d orbits\n", orbit());
}
--------------

To compile, just:

   dmd atom

and the pragma(root) will cause the compiler to behave as if you'd typed:

   dmd atom electron

Not only does this make for header-only libraries, but if you have a project 
that is split up into several modules, it becomes super convenient to compile
it 
if the other modules are all annotated with pragma(root).

Note that there's no "inline" anywhere. Inlining will remain about inlining,
and 
nothing else.
Jun 14
next sibling parent Tove <tove fransson.se> writes:
On Sunday, 14 June 2020 at 08:36:50 UTC, Walter Bright wrote:
 There is much misunderstanding in the "What does 'inline' 
 mean?" thread.

 I've slowly come to realize that it's not about inlining at 
 all, it's about a desire for "header-only" libraries. To use a 
 header-only library, you only have to import the module for it.

 This works today in D if you also add the imported module to 
 the command line. But, of course, it's good to make it simpler.

 So, I propose the addition of `pragma(root);` which looks like:

 --- electron.di ---
 module electron;
 pragma(root);    // header-only library!
I like this feature. Could *.di always be root?
Jun 14
prev sibling next sibling parent reply Max Samukha <maxsamukha gmail.com> writes:
On Sunday, 14 June 2020 at 08:36:50 UTC, Walter Bright wrote:

 To compile, just:

   dmd atom
We have -i already for automatic inclusion of imported modules: dmd atom -i=. What does the pragma add to this? I think Manu was asking for something different.
Jun 14
next sibling parent reply =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 6/14/20 2:33 AM, Max Samukha wrote:

 We have -i already for automatic inclusion of imported modules:
 
 dmd atom -i=.
 
 What does the pragma add to this?
-i would mean "assume everything has pragma(auto_add)". Note how I could not say "root". :) Where does it come from? Why root? Most programmers wouldn't understand it at all. "Header" wouldn't mean anything to anybody unless they are coming from C or C++. And you said "automatic inclusion". Perhaps pragma(auto_include) but then it would confuse C and C++ programmers because they "include" header files. Tough naming situation... :) Ali
Jun 14
parent Max Samukha <maxsamukha gmail.com> writes:
On Sunday, 14 June 2020 at 09:39:52 UTC, Ali Çehreli wrote:
 On 6/14/20 2:33 AM, Max Samukha wrote:

 We have -i already for automatic inclusion of imported modules:
 
 dmd atom -i=.
 
 What does the pragma add to this?
-i would mean "assume everything has pragma(auto_add)".
Yes. However, you can specify patterns as strict as matching single modules. Anyway, I can see the difference.
 Note how I could not say "root". :) Where does it come from? 
 Why root? Most programmers wouldn't understand it at all.
 "Header" wouldn't mean anything to anybody unless they are 
 coming from C or C++.

 And you said "automatic inclusion".
 Perhaps pragma(auto_include) but then it would confuse C and 
 C++ programmers because they "include" header files.

 Tough naming situation... :)
pragma(static) ))
 Ali
Jun 14
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/14/2020 2:33 AM, Max Samukha wrote:
 On Sunday, 14 June 2020 at 08:36:50 UTC, Walter Bright wrote:
 
 To compile, just:

   dmd atom
We have -i already for automatic inclusion of imported modules: dmd atom -i=. What does the pragma add to this? I think Manu was asking for something different.
I don't know what -i does exactly, or why it was added. https://dlang.org/dmd-windows.html#switch-i
Jun 14
next sibling parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Sunday, 14 June 2020 at 09:42:45 UTC, Walter Bright wrote:

 I don't know what -i does exactly, or why it was added.

 https://dlang.org/dmd-windows.html#switch-i
https://dlang.org/dmd-windows.html#switch-i[
.<
Jun 14
prev sibling next sibling parent reply Mathias LANG <geod24 gmail.com> writes:
On Sunday, 14 June 2020 at 09:42:45 UTC, Walter Bright wrote:
 On 6/14/2020 2:33 AM, Max Samukha wrote:
 On Sunday, 14 June 2020 at 08:36:50 UTC, Walter Bright wrote:
 
 To compile, just:

   dmd atom
We have -i already for automatic inclusion of imported modules: dmd atom -i=. What does the pragma add to this? I think Manu was asking for something different.
I don't know what -i does exactly, or why it was added. https://dlang.org/dmd-windows.html#switch-i
https://github.com/dlang/dmd/pull/7099 It's essentially putting rdmd in the compiler because rdmd was considered too slow.
Jun 14
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/14/2020 2:52 AM, Mathias LANG wrote:
 https://github.com/dlang/dmd/pull/7099
 It's essentially putting rdmd in the compiler because rdmd was considered too
slow.
Thank you. I see I was involved in the discussion. The issue is then should the implementer of the module decide if it is header-only (i.e. pragma(root)) or the caller (-i). It would probably be better as the implementer, as a header-only library has to be designed for it. It also means the user needn't need to set the switches, he can just compile.
Jun 14
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Sunday, 14 June 2020 at 10:24:33 UTC, Walter Bright wrote:
 On 6/14/2020 2:52 AM, Mathias LANG wrote:
 https://github.com/dlang/dmd/pull/7099
 It's essentially putting rdmd in the compiler because rdmd was 
 considered too slow.
Thank you. I see I was involved in the discussion. The issue is then should the implementer of the module decide if it is header-only (i.e. pragma(root)) or the caller (-i). It would probably be better as the implementer, as a header-only library has to be designed for it. It also means the user needn't need to set the switches, he can just compile.
I agree with everything except for the name. I same sure there are better names than root. Perhaps extern(static) ? Also this needs to be either restricted to leaf functions, or draw in callees as well.
Jun 14
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Sunday, 14 June 2020 at 10:35:51 UTC, Stefan Koch wrote:
 On Sunday, 14 June 2020 at 10:24:33 UTC, Walter Bright wrote:
 On 6/14/2020 2:52 AM, Mathias LANG wrote:
 https://github.com/dlang/dmd/pull/7099
 It's essentially putting rdmd in the compiler because rdmd 
 was considered too slow.
Thank you. I see I was involved in the discussion. The issue is then should the implementer of the module decide if it is header-only (i.e. pragma(root)) or the caller (-i). It would probably be better as the implementer, as a header-only library has to be designed for it. It also means the user needn't need to set the switches, he can just compile.
I agree with everything except for the name.
I take it back. I thought this was per_symbol rather than per_module. I would be on-board with doing this per symbol. And infact I am implementing it right now. To see where the challenges are.
Jun 14
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Sunday, 14 June 2020 at 10:24:33 UTC, Walter Bright wrote:
 It would probably be better as the implementer, as a 
 header-only library has to be designed for it.
That's not true. -i just works with almost any D code. It is enormously useful exactly the way it is.
Jun 14
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/14/20 11:36 AM, Adam D. Ruppe wrote:
 On Sunday, 14 June 2020 at 10:24:33 UTC, Walter Bright wrote:
 It would probably be better as the implementer, as a header-only 
 library has to be designed for it.
That's not true. -i just works with almost any D code. It is enormously useful exactly the way it is.
What code wouldn't it work with? Would be good to document those. "How to write a header-only D library". Blog post!
Jun 14
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Sunday, 14 June 2020 at 16:05:39 UTC, Andrei Alexandrescu 
wrote:
 What code wouldn't it work with?
It isn't so much code as it is some build paradigms. If you are forced to do separate compilation (like if dmd is eating 30 GB of RAM...), you can't enjoy dmd -i just working. But you can still use it in individual pieces then link the result together normally, so it doesn't get in your way, it just doesn't help as much. If you do it wrong in these cases, you are liable to get duplicate symbol problems with a diamond dependency (two packages compiled separately that both include a common third package). All manageable and -i can still help in these cases with appropriate inclusion/exclusion patterns, just -i's biggest strength is when you are doing an all-at-once build. Ditto with non-module based dependency versions or compilation flags, since those again bring you back to separate compilation. The other semi-tricky thing is dub's default source layout doesn't follow the convention -i expects due to having the extra source/ directory. So you need to add it with -Iproject/source each time. Or you can automatically rearrange files from a package upon downloading it (just read the `module` declaration and sort it into directories from that under a common `lib` top-level thing. This is actually what I do on my computer and why my arsd repo has most its files at top-level - if you clone the repo in your lib dir, the files are now right at `arsd/cgi.d` etc... which automatically works with the compiler. Zero extra config needed, it all just works. I get a lot of complaints about that layout but this is an objective advantage I don't think a lot of people really realize!) I've been tempted before to write a competing package manager that does stuff like this all automatically. Imagine you just write some D and `import foo.bar;` then the compiler (or, more specifically, the helper program that works with the compiler) tells you "module foo.bar is not installed but can be found in package `footech`, install now? (y/n)". And if you made the major version part of the module names instead of git tags, that'd all magically just work too. I'd be kinda fun and very efficient thanks to dmd -i.
Jun 14
parent reply Mathias LANG <geod24 gmail.com> writes:
On Monday, 15 June 2020 at 00:55:59 UTC, Adam D. Ruppe wrote:
 The other semi-tricky thing is dub's default source layout 
 doesn't follow the convention -i expects due to having the 
 extra source/ directory. So you need to add it with 
 -Iproject/source each time. Or you can automatically rearrange 
 files from a package upon downloading it (just read the 
 `module` declaration and sort it into directories from that 
 under a common `lib` top-level thing. This is actually what I 
 do on my computer and why my arsd repo has most its files at 
 top-level - if you clone the repo in your lib dir, the files 
 are now right at `arsd/cgi.d` etc... which automatically works 
 with the compiler. Zero extra config needed, it all just works. 
 I get a lot of complaints about that layout but this is an 
 objective advantage I don't think a lot of people really 
 realize!)
On the other hand, you rely on the person checking out the repository with a specific directory name, which is IMO much worse than the dub approach. If I check it out as "arsd-v1.1.1", suddenly it breaks, unless I use two DMD flags to solve the situation (-I and -mv). Additionally, many people want to separate their code and their diet files / config file / scripts / whatever and having the project's source file at the root make this impossible. So yeah I'll take the dub approach over this any day of the week, I just would like dub to generate source/projectname/ and not just source/
Jun 16
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Tuesday, 16 June 2020 at 13:27:00 UTC, Mathias LANG wrote:
 If I check it out as "arsd-v1.1.1", suddenly it breaks, unless 
 I use two DMD flags to solve the situation (-I and -mv).
Which is required in ALL cases by the dub approach, you just don't notice it as much because `dub build` sets all that based on the settings in the dub config file (or their conventional defaults). But you lose something there: optional modules are compiled in by dub (unless you go through the config process to separate it), whereas with dmd -i they would not be processed at all. It is also impossible to mix versions of dependencies with the dub approach; you can rename that directory to -v1.1.1 but put in a -v1.2.1 next to it and you just have trouble (which is why dub resolves it down to just the one compatible version, or fails if it can't find one...) Maybe one of these days I'll actually write my competing package manager and demo all this stuff (you wouldn't even have to change your layouts btw since it is easy to auto-scan packages for D modules and rearrange them as needed; my doc site does exactly this). But since I don't use packages it isn't a big priority of mine.
Jun 16
parent reply Mathias LANG <geod24 gmail.com> writes:
On Wednesday, 17 June 2020 at 02:51:20 UTC, Adam D. Ruppe wrote:
 On Tuesday, 16 June 2020 at 13:27:00 UTC, Mathias LANG wrote:
 If I check it out as "arsd-v1.1.1", suddenly it breaks, unless 
 I use two DMD flags to solve the situation (-I and -mv).
Which is required in ALL cases by the dub approach, you just don't notice it as much because `dub build` sets all that based on the settings in the dub config file (or their conventional defaults).
Not quite, only `-I` is required, not `-mv`, which is fair IMO, and expected from anyone familiar with C[++].
 But you lose something there: optional modules are compiled in 
 by dub (unless you go through the config process to separate 
 it), whereas with dmd -i they would not be processed at all.
I totally agree that the current situation on that regard is terrible, although that's more of a dub limitation than something inherent to the hierarchy. `dub` has a concept of `mainSourceFile` and I hope one day it'll use this to drive compilation instead of selecting all files. I have a somewhat related bug report: https://github.com/dlang/dub/issues/1778
 It is also impossible to mix versions of dependencies with the 
 dub approach; you can rename that directory to -v1.1.1 but put 
 in a -v1.2.1 next to it and you just have trouble (which is why 
 dub resolves it down to just the one compatible version, or 
 fails if it can't find one...)
I don't quite see how that relates to the discussion. I was merely stating that relying on the external name of the directory was much worse than having the user pass `-Iproject_name/source` by command line. Nonetheless, I don't see why you would ever want to mix two semver-incompatible versions of the same library. I get that it can happen due to transitive dependencies, but there's really no way to make it foolproof for non-trivial cases, and I think everyone's time and energy are best spent avoiding this situation than catering for it.
 Maybe one of these days I'll actually write my competing 
 package manager and demo all this stuff (you wouldn't even have 
 to change your layouts btw since it is easy to auto-scan 
 packages for D modules and rearrange them as needed; my doc 
 site does exactly this).
Contributions to `dub` are welcome, will be immediately available to users, and will benefit the community much more than yet another package manager (we have `dub`, `dud`, `reggae`, and probably a plethora of other tools).
Jun 16
parent Martin Tschierschke <mt smartdolphin.de> writes:
On Wednesday, 17 June 2020 at 03:21:30 UTC, Mathias LANG wrote:
[...]
 Contributions to `dub` are welcome, will be immediately 
 available to users, and will benefit the community much more 
 than yet another package manager (we have `dub`, `dud`, 
 `reggae`, and probably a plethora of other tools).
+1 !
Jun 17
prev sibling parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Sunday, 14 June 2020 at 16:05:39 UTC, Andrei Alexandrescu 
wrote:
 On 6/14/20 11:36 AM, Adam D. Ruppe wrote:
 On Sunday, 14 June 2020 at 10:24:33 UTC, Walter Bright wrote:
 It would probably be better as the implementer, as a 
 header-only library has to be designed for it.
That's not true. -i just works with almost any D code. It is enormously useful exactly the way it is.
What code wouldn't it work with? Would be good to document those. "How to write a header-only D library". Blog post!
The '-i' flag is the symmetric opposite of '-I'. '-I' expects either header-only libraries, or separate compilation, while '-i' expects neither. What Walter is suggesting (marking modules with 'pragma (root);' to make them header-only mainly serves the '-I' flag. The only advantage 'pragma (root);' may have to '-i' is a compile-time guarantee that modules intended to be header-only actually don't impose any link-time dependencies. The main use case of the `-i` flag is to compile (not just import) modules. However the killer feature of '-i' is that while '-I' a is blunt tool that just adds all modules under a dir to the import path, '-i' leverages D's import system and lazily compiles only the modules that are imported starting from the root set of modules. For example, giving the following project structure: project/ ├── lib1 │ ├── mod1.d │ ├── ... │ └── modn.d ├── lib2 │ ├── mod1.d │ ├── ... │ └── modn.d └── main.d Previously, in order to compile it, you could do the following: find lib1/ lib2/ -type f -name '*.d' | xargs dmd -betterC main.d However, this has the disadvantage of compiling stuff that main.d doesn't need. Say main.d was this: extern (C) int main(int argc, const char** argv) { import lib1.mod1 : fun1a; return fun1a(argc, argc + 1); } The only function that ever needs to be compiled is fun1a. However the command above will end up compiling the whole world (every module under lib1/). With the '-i' flag we can compile main.d like so: dmd -betterC -i=lib1 -i=lib2 main.d Now, set of modules that would be compiled equals precisely the set of modules imported from main.d - in this case only lib1/mod1.d. The only deficiency currently is that it doesn't take advantage of selective imports - each imported module is compiled as a whole, even if you only need to compile the imported symbols. This problem is not just an implementation deficiency though, as whether something is compiled and whether it exists is somewhat conflated and has impact on D's meta programming (DbI and all that). I think the best way to address this issue is at the root - we need to define precise semantics over what lazy compilation means. I have been meaning to write a DIP that addresses this issue for a while, based on the idea of an export attribute (orthogonal to public/protected/package/private access modifiers) and export inference. You can see a sketch of what I have in mind in these posts: https://forum.dlang.org/post/xbllqrpvflazfpowizwj forum.dlang.org https://forum.dlang.org/post/nvvgrdlucajshjngdjlj forum.dlang.org
Jun 16
parent Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Tuesday, 16 June 2020 at 07:06:16 UTC, Petar Kirov 
[ZombineDev] wrote:
 The '-i' flag is the symmetric opposite of '-I'. '-I' expects 
 either header-only libraries, or separate compilation, while 
 '-i' expects neither.
I meant: The '-i' flag is the symmetric opposite of '-I'. '-I' requires either header-only libraries, or separate compilation, while '-i' *requires neither*.
Jun 16
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/14/2020 8:36 AM, Adam D. Ruppe wrote:
 On Sunday, 14 June 2020 at 10:24:33 UTC, Walter Bright wrote:
 It would probably be better as the implementer, as a header-only library has 
 to be designed for it.
That's not true. -i just works with almost any D code. It is enormously useful exactly the way it is.
If you're building a project using separate compilation, I doubt you'd want an import with large functions in it being compiled over and over into multiple object files. Also, you probably don't want dependencies leaking into the imported code.
Jun 14
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 15 June 2020 at 00:17:42 UTC, Walter Bright wrote:
 If you're building a project using separate compilation, I 
 doubt you'd want an import with large functions in it being 
 compiled over and over into multiple object files.
If you're using separate compilation, you won't pass -i. You might pass -i=-foo.bar though; the optional pattern part of the argument is pretty flexible to pick and choose what you do and do not want.
Jun 14
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Sunday, 14 June 2020 at 09:42:45 UTC, Walter Bright wrote:
 I don't know what -i does exactly, or why it was added.
-i is the most exciting thing that happened in the compiler from like 2015 up to 2018. Enormously useful in efficient, seamless builds, serious gamechanger in using D. Don't touch it!
Jun 14
next sibling parent reply Seb <seb wilzba.ch> writes:
On Sunday, 14 June 2020 at 15:31:05 UTC, Adam D. Ruppe wrote:
 On Sunday, 14 June 2020 at 09:42:45 UTC, Walter Bright wrote:
 I don't know what -i does exactly, or why it was added.
-i is the most exciting thing that happened in the compiler from like 2015 up to 2018. Enormously useful in efficient, seamless builds, serious gamechanger in using D. Don't touch it!
I fully agree, but there's a very sad side-story to it: https://github.com/dlang/tools/pull/271 https://github.com/dlang/tools/pull/290 tl;dr: rund [1] (a modern variant of rdmd without the compile twice bug + useful features like including flags inside the file header) was a result of this revert debacle. At this point rund is vastly superior to rdmd and there's almost zero talk about it (both: the rdmd debacle + rund). [1] https://github.com/dragon-lang/rund
Jun 14
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Sunday, 14 June 2020 at 16:19:54 UTC, Seb wrote:
 I fully agree, but there's a very sad side-story to it:
Indeed, rund is a practical piece of work that actually adds value over dmd -i for some cases. Why was that rdmd change reverted? Is it just because the other compiler versions in repos didn't support -i?
Jun 14
prev sibling next sibling parent Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Sunday, 14 June 2020 at 15:31:05 UTC, Adam D. Ruppe wrote:
 On Sunday, 14 June 2020 at 09:42:45 UTC, Walter Bright wrote:
 I don't know what -i does exactly, or why it was added.
-i is the most exciting thing that happened in the compiler from like 2015 up to 2018. Enormously useful in efficient, seamless builds, serious gamechanger in using D. Don't touch it!
+1000 That flags is the workhorse of all our builds in the company: don't touch it .... please /Paolo
Jun 14
prev sibling parent novice2 <sorry noem.ail> writes:
On Sunday, 14 June 2020 at 15:31:05 UTC, Adam D. Ruppe wrote:
 -i is the most exciting thing that happened in the compiler 
 from like 2015 up to 2018. Enormously useful in efficient, 
 seamless builds, serious gamechanger in using D.

 Don't touch it!
+100500
Jun 16
prev sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Sun, 14 Jun 2020 01:36:50 -0700 schrieb Walter Bright:


 and the pragma(root) will cause the compiler to behave as if you'd
 typed:
 
    dmd atom electron
If it really does only that, it'll lead to duplicate symbol errors as soon as electron introces any symbols which do not have weak linkage. This includes global variables, but that may be expected not to work in header-only libraries. This problem however also occurs because of initZ, TypeInfo and other special symbol generated for structs and classes. As mentioned in the other post, even if you don't declare structs or classes, right now this does not work because of duplicate ModuleInfo. But those are implementation issues. Regarding the feature: I don't really like the name "root". I also wonder whether it would not be better to unify all "emit to object file" use cases into one pragma: // CTFE only function (no codegen) pragma(emit, never) void foo() {} // can be used at runtime (codegen) but never output a symbol pragma(emit, inliner) // Only once, in the defining module / CU pragma(emit, normal) // Once into each into CUs importing this. Not into defining CU pragma(emit, imports) // Emit into all importing and defining CU pragma(emit, all) It should be noted that due to D's name mangling, whenever symbols are emitted multiple times (imports, all) they have to be weak and will be merged by linker, so you never get multiple copies. If a static inline effect is desired (emit multiple copies with independent addresses), we'd need pragma(emit, all_copy) and pragma(import_copy). However, I'm not sure if this should be part of pragma(emit), or an additional orthogonal pragma. -- Johannes
Jun 14
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Sunday, 14 June 2020 at 16:42:36 UTC, Johannes Pfau wrote:
 Am Sun, 14 Jun 2020 01:36:50 -0700 schrieb Walter Bright:


    [...]
If it really does only that, it'll lead to duplicate symbol errors as soon as electron introces any symbols which do not have weak linkage. This includes global variables, but that may be expected not to work in header-only libraries. [...]
Proof of concept here: https://github.com/dlang/dmd/pull/11272
Jun 14