www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - reggae v0.10.0 - The meta build system just got better

reply Atila Neves <atila.neves gmail.com> writes:
https://code.dlang.org/packages/reggae

For those who don't know, reggae is a meta-build system for and 
in D. It's like CMake, if you replace their terrible language 
with D*. Like CMake, it can output make and ninja files. Unlike 
CMake, it can also output tup files and has its own binary 
generator in case the dependency on one of those programs is 
undesired. Also unlike CMake, it supports dub projects 
out-of-the-box by using dub as a library.

TL;DR: you can use to build your dub project and it'll be faster 
and not rebuild the whole thing if you only change one module**.

Highlights of this release:

When dinosaurs roamed the Earth, `dmd -makedeps` didn't work so 
to automatically handle dependencies between D modules reggae had 
to wrap the compiler with a binary named `dcompile`. This would 
call the compiler with `-v`, get the imports from the output, and 
use them to only rebuild what was necessary. Now that `dmd 
-makedeps` got fixed, the extra complication (and time to build 
the extra binary) is gone.

Default dub builds are much faster now. Before, reggae would 
create a reggaefile.d if one didn't exist for dub projects. I got 
tired of waiting for the compiler, and realised that I don't 
actually have to compile D code on-the-fly if I know ahead of 
time what the code is anyway. Now, reggae immediately*** spits 
out the build files for such projects.

Automatically detect when new files are added/deleted: this is 
the main reason that the CMake devs erroneously advise people to 
not use "globs" to just "give me all the source files". The issue 
is when the file list changes, you need to remember to rerun the 
metabuild system to detect the filesystem changes. No more: 
reggae builds now rerun reggae when files are added or deleted. 
Tup, of course, has always done this but it's the only one I know 
of that did.

Before, if the compiler binary changed, then... oops it was if 
you rebuilt in between updating your distro. Now the object files 
depend on the compiler binary.



* Or Python, Ruby, Javascript, or Lua.
** Unless you made every module depend on every other module, in 
which case, don't do that.
** dub can be slow to start up. Slows for me is ~200ms.
Sep 07
next sibling parent reply German Diago <germandiag gmail.com> writes:
On Thursday, 7 September 2023 at 17:34:48 UTC, Atila Neves wrote:
 https://code.dlang.org/packages/reggae

 For those who don't know, reggae is a meta-build system for and 
 in D. It's like CMake...
How mature is the build system? FWIW, I would like to stay in Meson if I can, since I use C++ also and that makes combining libs and other niceties easier.
Sep 07
parent Atila Neves <atila.neves gmail.com> writes:
On Thursday, 7 September 2023 at 21:25:27 UTC, German Diago wrote:
 On Thursday, 7 September 2023 at 17:34:48 UTC, Atila Neves 
 wrote:
 https://code.dlang.org/packages/reggae

 For those who don't know, reggae is a meta-build system for 
 and in D. It's like CMake...
How mature is the build system? FWIW, I would like to stay in Meson if I can, since I use C++ also and that makes combining libs and other niceties easier.
Mature enough that we've been using it at work for years now, but YMMV depending on what you use it for. It also supports C++ BTW.
Sep 07
prev sibling next sibling parent Anonymouse <zorael gmail.com> writes:
On Thursday, 7 September 2023 at 17:34:48 UTC, Atila Neves wrote:
 [...]
Can reggae handle non-trivial dub builds with trees of dependencies? I gave it a quick test, following the examples in the readme, and the Makefile it generated looked for sources in the wrong place. It also didn't build the dependencies (nor even include their source directories as import paths). Maybe I'm doing it wrong.
Sep 09
prev sibling next sibling parent Greggor <Greggor notareal.email> writes:
On Thursday, 7 September 2023 at 17:34:48 UTC, Atila Neves wrote:
 https://code.dlang.org/packages/reggae

 For those who don't know, reggae is a meta-build system for and 
 in D. It's like CMake, if you replace their terrible language 
 with D*. Like CMake, it can output make and ninja files. Unlike 
 CMake, it can also output tup files and has its own binary 
 generator in case the dependency on one of those programs is 
 undesired. Also unlike CMake, it supports dub projects 
 out-of-the-box by using dub as a library.

 [...]
I spent a some time getting a part of my project to build with Reggae & so far I am impressed, The partial compiles are really good, I am looking forward to switching! My largest complaint is that Reggae needs improvements to it's documentation, there was enough of it for me to get started and I can figure it out, but I do not feel that anyone can just jump right into using it. But this is not a unique problem especially with build systems, so this is a nitpick.
Sep 11
prev sibling next sibling parent reply Adam D Ruppe <destructionator gmail.com> writes:
I tried reggae today. It did not go well.

http://dpldocs.info/this-week-in-d/Blog.Posted_2023_09_11.html#reggae-editorial
Sep 11
parent reply Atila Neves <atila.neves gmail.com> writes:
On Monday, 11 September 2023 at 23:15:28 UTC, Adam D Ruppe wrote:
 I tried reggae today. It did not go well.

 http://dpldocs.info/this-week-in-d/Blog.Posted_2023_09_11.html#reggae-editorial
One thing I learned from you is that for small projects `dmd -i` is just as fast as incremental compilation, which is rather surprising to me. It does mean adding `-I` flags to every dependency though, so there's that. And for larger codebases `dmd -i` can't and won't work.
Sep 12
parent reply Adam D Ruppe <destructionator gmail.com> writes:
On Tuesday, 12 September 2023 at 13:12:29 UTC, Atila Neves wrote:
 It does mean adding `-I` flags to every dependency though, so 
 there's that.
Not if you install them properly.
 And for larger codebases `dmd -i` can't and won't work.
Define "larger".
Sep 12
parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 12 September 2023 at 13:17:50 UTC, Adam D Ruppe wrote:
 On Tuesday, 12 September 2023 at 13:12:29 UTC, Atila Neves 
 wrote:
 It does mean adding `-I` flags to every dependency though, so 
 there's that.
Not if you install them properly.
That's the job of the package manager does that. An argument could be made that it could/should install the dependencies such that only one `-I` flag is needed. Even then, I'm not sure how it'd handle different versions of the same package unless it did something like what Ikey showed at DConf.
 And for larger codebases `dmd -i` can't and won't work.
Define "larger".
~190k SLOC (not counting the many dub dependencies) killed dmd on a system with 64GB RAM + 64GB swap after over a minute. Even if it worked, it'd be much, much slower.
Sep 15
parent Adam D Ruppe <destructionator gmail.com> writes:
On Friday, 15 September 2023 at 20:22:50 UTC, Atila Neves wrote:
 An argument could be made that it could/should install the 
 dependencies such that only one `-I` flag is needed.
Indeed, this would be god tier.
 ~190k SLOC (not counting the many dub dependencies) killed dmd 
 on a system with 64GB RAM + 64GB swap after over a minute. Even 
 if it worked, it'd be much, much slower.
What you do with the lines of code is *far* more important than how many there are. The arsd library has about 219,000 lines of text if you delete the Windows-only and obsolete modules (doing so just so I can actually dmd *.d here on my Linux box). This includes comments and such; dscanner --sloc reports about 98,000. $ wc *.d <snip> 218983 870208 7134770 total $ dscanner --sloc *.d <snip> total: 98645 Let's compile it all: $ /usr/bin/time dmd *.d -L-L/usr/local/pgsql/lib -unittest -L-lX11 5.35user 0.72system 0:06.08elapsed 99%CPU (0avgtext+0avgdata 1852460maxresident)k 0inputs+70464outputs (0major+536358minor)pagefaults 0swaps That's a little bit slow, over 5 seconds. About 1.3 of those seconds are spent in the linker, the other 4 remain with dmd -c. Similarly, that's almost 2 GB of RAM it used, more than it probably should, but it worked fine. My computer btw is a budget model circa 2016. Nothing extraordinary about its hardware. But notice it isn't actually running out of RAM or melting the CPU over a period of minutes, despite being about six figures lines of code but any measure. On the other hand, compile: enum a = () { string s; foreach(i; 0 .. 20_000_000_000) s ~= 'a'; return s; }(); Don't actually do it, but you can imagine what will happen. 6 lines that can spin your cpu and explode your memory. Indeed, even just importing this module, even if the build system tried not to compile it again, will cause the same problem. The arsd libs are written - for the most part, there's some exceptions - with compile speed in mind. If I see my build slow down, I investigate why. Most problems like this can be fixed! In fact, let's take that snippet and talk about it. I had to remove *several* zeroes to make it even work without freezing up my computer, but with a 100,000 item loop, it just barely worked. Even 200,000 made it OOM. But ok, a 100,000 item append: 0.53user 1.52system 0:02.17elapsed 95%CPU (0avgtext+0avgdata 4912656maxresident)k About 5 GB of RAM devoured by these few lines, taking 2 seconds to run. What are some ways we can fix this? The ~= operator is actually *awful* at CTFE, its behavior is quadratic (...or worse, i didn't confirm this today, but it is obviously bad). So you can fix this pretty easily: enum string a = () { // preallocate the buffer instead of append char[] s = new char[](100000); foreach(ref ch; s) ch = 'a'; return s; }(); 0.17user 0.03system 0:00.21elapsed 98%CPU (0avgtext+0avgdata 45748maxresident)k 16inputs+1408outputs (0major+21995minor)pagefaults 0swaps Over 10x faster to compile, 1/100th of the RAM, ram result. Real world code is frequently doing more than this example and rewriting it to work like this might take some real effort.... but the results are worth it. And btw try this: import this module and check your time/memory stats. Even if it isn't compiled, since ctfe is run when the module is even just imported, you gain *nothing* by separate compilation! ...but there are times when you can gain a LOT by separate compilation in situations like this, if you can move the ctfe to be some private thing not exposed in the interface. This requires some work by the lib author too though in most cases. An example where you can gain a lot is when something does a lot of internal code generation but exposes a small interface, for example a scripting language wrapper (though script wrappers can also be made to compile reasonably efficiently if you use things like preallocation of buffers, keep your generated functions short (again, the codegen has quadratic behavior, so many small functions work better than a big one, and if you factor the code well, you can minimize the amount of generated code and call back to generic things, e.g. type erasure), collapse template instances, and keep ctfe things ctfe only with a variety of techniques, so they are not codegened unless they are actually necessary). My arsd.script and arsd.cgi can wrap large numbers of functions and classes reasonably well, but that's why programs using them tend to be multi-second builds.... just note that's programs using them. Separate compiling the libraries doesn't help. You'd have to structure the code to keep those codegen parts internal to a package with a minimal interface, then separate compiling those internal components might win. But this is a fairly niche case. Yes, I know there's one major commercial D user who do exactly this. But that's the exception, not the rule.
Sep 17
prev sibling parent zjh <fqbqrr 163.com> writes:
On Thursday, 7 September 2023 at 17:34:48 UTC, Atila Neves wrote:
 https://code.dlang.org/packages/reggae
You can use professional `xmake`, It's very convenient.
Sep 11