www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - [Semi-OT] On the topic of build tools

reply SealabJaster <sealabjaster gmail.com> writes:
This is a rather aimless discussion, but I wanted to get this off 
my mind since I've been hopping around different build systems 
recently. Also it's 2am.

I've recently started to move out of my shell of cushy build 
systems managed by an IDE, or super simple ones like Dub, and 
have tried out a few different build tools now:

* CMake
* Scons
* Meson
* Bazel

Cmake: I like it when I'm not the one writing it. Its language 
feels like it has the same backwards compat and weird syntax 
issues that C++ deals with, which I found kind of ironic.

Scons: I don't know why but I couldn't get my head around it. I 
found it confusing where I simply wanted to include a simple D 
library into a simple D app.

Meson: I actually kind of like Meson because it seems to focus on 
being a build tool and not a complete language + dep manager + 
everything and the sink. It does feel a bit rigid, and the D 
support is iffy (I still can't figure out how to make the auto 
dub config spit out dub dependencies), but was probably the most 
useable one for D.

Bazel: I'm still mentally recovering.

My questions are simple, but I fear the answers are far, far from 
it:

* Are there any build systems are aren't... 'awful', 'anti-fun', 
etc?

* Build tools for some reason decide that they should also be 
pseudo or actual package managers. How much could a realistic 
tool be decoupled from having to manage dependencies (it'd of 
course need to know about subprojects and the like).

* What would the ideal build tool look like, and be able to do? 
Especially in regards to its configuration language.

* It appears to be very difficult to make a build tool that isn't 
just `make` with a pretty coat of paint?

Some tools feel so rigid and inflexible, even if you stay within 
bounds of its supported languages, whereas other tools feel far 
too 'free' in a sense that there's no clear direction on how to 
do certain things, even if they're technically possible.

As my autism-ridden brain loves to do, it got rather obsessed 
with this topic for a while, and I simply wanted to see what the 
general feelings on these tools are on this forum, because all 
the tools seem to be "necessary evils" rather than "gifts from 
god" in terms of their usage and popularity compared to their 
difficulties and downfalls.

I find Bazel's approach quite interesting: being able to download 
tools and language support straight from the same language that 
configures your builds. I believe language support is also added 
via said language.

I found the idea of using two languages interesting: e.g. use 
something like LUA to defines tooling, how to build X files, 
etc., then something more constrained and declarative like SDL to 
make use of the things the backend language provides. So 
essentially a seperation of concerns between the extension and 
configuration languages.

Sorry if this is a bit incoherent, but I'm finding it kind of 
hard to express what my thoughts are on the topic. I guess my 
main desire is to simply start a discussion on what potential a 
'proper' build tool could have compared to what's currently out 
there.

I've seen previous discussions before about separation between 
building, dependency management, dependency+tool fetching, etc. 
and it feels like an interesting topic for how 'bad', fractured, 
and so on the actual tooling is.

Also: https://xkcd.com/927/

Apologies for the faff, but I literally feel an urge to post this 
for some reason, and it's difficult to resist that feeling.

(also p.s., as I tend to do, I'll wake up tomorrow and feel too 
scared to reply, so I'll definitely see any replies, but may not 
respond. Sorry!)
Aug 09
next sibling parent Tejas <notrealemail gmail.com> writes:
On Tuesday, 10 August 2021 at 01:11:45 UTC, SealabJaster wrote:
 This is a rather aimless discussion, but I wanted to get this 
 off my mind since I've been hopping around different build 
 systems recently. Also it's 2am.

 [...]
If someone us really going to read, please also help in determining whether Microsoft's vcpkg can also be used for building D projects. Thanks in advance.
Aug 10
prev sibling next sibling parent Dukc <ajieskola gmail.com> writes:
On Tuesday, 10 August 2021 at 01:11:45 UTC, SealabJaster wrote:
 * Are there any build systems are aren't... 'awful', 
 'anti-fun', etc?
Quickly reading, it seems that you might like [Reggae](https://code.dlang.org/packages/reggae)
 * Build tools for some reason decide that they should also be 
 pseudo or actual package managers. How much could a realistic 
 tool be decoupled from having to manage dependencies (it'd of 
 course need to know about subprojects and the like).
The ones that I know and do so (DUB and Nix) can both be configured to use local directories as dependencies if you prefer to fetch stuff manually or with another package manager.
Aug 10
prev sibling next sibling parent Denis Feklushkin <denis.feklushin gmail.com> writes:
On Tuesday, 10 August 2021 at 01:11:45 UTC, SealabJaster wrote:
 * Meson
Pros: * It is hard to write working but unmaintainable scripts. Language constructed by way that Meson config files isn't evolve into spaghetti. * Many popular projects now move to Meson * Doesn't use (even if you really want to) the source tree for storing intermediate files or resulting binaries * Language independent * Easier to learn than cmake * Active development, many commits merged each day Cons: * Harder to learn than DUB * Only few mesonified D packages * A lot of D-related bugs * Still need to understanding what really goes when you compile something, this isn't silver bullet
Aug 10
prev sibling next sibling parent reply Domain <dont_email empty.com> writes:
On Tuesday, 10 August 2021 at 01:11:45 UTC, SealabJaster wrote:
 This is a rather aimless discussion, but I wanted to get this 
 off my mind since I've been hopping around different build 
 systems recently. Also it's 2am.

 [...]
https://github.com/xmake-io/xmake You can try xmake.
Aug 10
parent evilrat <evilrat666 gmail.com> writes:
On Wednesday, 11 August 2021 at 00:02:38 UTC, Domain wrote:
 On Tuesday, 10 August 2021 at 01:11:45 UTC, SealabJaster wrote:
 This is a rather aimless discussion, but I wanted to get this 
 off my mind since I've been hopping around different build 
 systems recently. Also it's 2am.

 [...]
https://github.com/xmake-io/xmake You can try xmake.
Looks nice and slim, but... I see too much stringly typed stuff using scripting language, not sure if it'll work for me over CMake even in mixed C++/D projects. Also what catches my eyes is use of target() declaration and nested block, looks fishy to me.
Aug 12
prev sibling next sibling parent reply ted <spam spam.com> writes:
On 10/8/21 10:41 am, SealabJaster wrote:
 This is a rather aimless discussion, but I wanted to get this off my 
 mind since I've been hopping around different build systems recently. 
 Also it's 2am.
 
Have a look at: https://github.com/GrahamStJack/bottom-up-build Used in significant projects with both C++ and D involved. Handles code generated during a build (& its dependencies). Fairly simple configuration/setup. Enforces simple dependencies (will refuse to build if there are circular dependencies).
Aug 11
parent SealabJaster <sealabjaster gmail.com> writes:
Several people wrote:
 ...
Thank you all for your suggestions, XMake especially seems quite interesting to try out.
 A lot of D-related bugs
_oh yes there are_, regardless though, I think Meson is currently my personal favourite. For the last few days I've been toying with the idea of "what would my own build tool look like?" and even hacked together something. But what I'm finding is: it's quite difficult to design something cohesive; simple by default yet powerful when opted-in, and just in general is useful enough to be worth using instead of already existing tools (who would've thought). My lack of experience certainly doesn't help either >x) Maybe there's a valid reason 200 different tools exist...
Aug 11
prev sibling next sibling parent reply ruki <waruqi gmail.com> writes:
On Tuesday, 10 August 2021 at 01:11:45 UTC, SealabJaster wrote:
 This is a rather aimless discussion, but I wanted to get this 
 off my mind since I've been hopping around different build 
 systems recently. Also it's 2am.

 [...]
xmake can build a mixed c/dlang project, and supports built-in package dependency management. It also supports downloading integrated dlang modules from dub. For example add_requires("dub::zlib") https://github.com/xmake-io/xmake/blob/test/tests/projects/dlang/console_with_pkgs/xmake.lua
Aug 13
parent reply SealabJaster <sealabjaster gmail.com> writes:
On Friday, 13 August 2021 at 14:08:29 UTC, ruki wrote:
 xmake can build a mixed c/dlang project, and supports built-in 
 package dependency management. It also supports downloading 
 integrated dlang modules from dub. For example 
 add_requires("dub::zlib")
Xmake is looking more and more interesting >:D
 imperative scripts, which generate dependency information as a 
 side effect
This was kind of what my hackish experiment was going for. You'd define things in LUA that manually called "isOutOfDate(file, {metadata})" and "update(file, {metadata})", and then the SDL half was just a list of function calls (quite literally since the SDL was translated into LUA). Another pretty OT question, but does anyone else have this ridiculous urge to make their own versions of everything? Like, instead of being sensible and using an already existing thing, you instead want to make a crappier, buggier, unfinished version simply because its "mine".
Aug 13
next sibling parent Mike Parker <aldacron gmail.com> writes:
On Friday, 13 August 2021 at 19:30:29 UTC, SealabJaster wrote:

 This was kind of what my hackish experiment was going for. 
 You'd define things in LUA that manually called 
 "isOutOfDate(file, {metadata})" and "update(file, {metadata})", 
 and then the SDL half was just a list of function calls (quite 
 literally since the SDL was translated into LUA).
Premake uses Lua: https://premake.github.io/ Manu got D support into it quite a while back. Not sure where it stands now.
Aug 13
prev sibling parent reply evilrat <evilrat666 gmail.com> writes:
On Friday, 13 August 2021 at 19:30:29 UTC, SealabJaster wrote:
 Another pretty OT question, but does anyone else have this 
 ridiculous urge to make their own versions of everything?

 Like, instead of being sensible and using an already existing 
 thing, you instead want to make a crappier, buggier, unfinished 
 version simply because its "mine".
I do, a lot of people have that issue as well. This isn't really an issue until it is controlled, but when it's not the end result is almost empty GH page and a broken career. Every time I need to start a project I have this: "meh, these libraries sucks, the author was stupid", and it is easy to let it loose like: "ugh, the tooling is bloat and buggy, I'll make my own IDE quickly", and then in some rare cases it evolves: "this OS is junk, if I am to move forward I will need my own OS". So what is the deal with it? Well, when you rejected all libraries for the task (especially for reasons like style) it is time to stop, take a deep breath and swallow (your ego), remember - you have things to do, and if it is a paid job they pays you for that specific things and not for reinventing the wheel. Additionally you will have deadlines, and you had to support that thing afterwards for undetermined time. Would you really like supporting all your inventions for a lifetime? Probably not. And then if you just started to make that replacement for a specific library when you started to plan things and implementing it you will definitely hit some issues related to subject that simply wasn't there when you were planning, and this is where you probably start to understand why authors of those "stupid" libraries did "stupid" things. At this point it is time to stop doing crap, turn down your pride, thank the authors for saved time, and start using the said libraries. Enjoy your life with much less stress and depression! Now, instead of "libraries" you could as well put "software" and it will be the same, you can even put "hardware" and it still will be the same. And that is how it works! Just remember - when you see "junk" or "bloat" it well could be the author's poor skills or incompetency (everyone hate amateurs, eh?), or it could be a real problem that was solved using what was really objectively possible at that moment because no one else did that before (or at least doesn't bothered to share, can't blame them though). And the latter is much more possible, unless of course it's just 300 lines of code prototype made at Friday's evening.
Aug 13
next sibling parent Dukc <ajieskola gmail.com> writes:
On Saturday, 14 August 2021 at 06:27:54 UTC, evilrat wrote:
 I do, a lot of people have that issue as well. This isn't 
 really an issue until it is controlled, but when it's not the 
 end result is almost empty GH page and a broken career.
Might I add that there is a middle-road way to deal with this. One obviously should not develop everything from ground up, but I think it's okay to apply small personal patches to editors, version control or compilers, for example. Also for simple stuff, like build scripts or syntax colouring schemes, it's totally reasonable to write your own if that works better. Of course, you should always mind compatibility with others when doing this. If you tweak your DMD for example, I think it's reasonable to make tweaks to it's error reporting format, but not to the language semantics - otherwise you are not writing D anymore. The latter can be done if you're not expecting anybody else to work with your code.
 Every time I need to start a project I have this:

   "meh, these libraries sucks, the author was stupid",
Guess where I though kind-of like that? When reading the API of C standard library! https://cplusplus.com/reference/cstdlib/atof/
 If no valid conversion could be performed, the function
 returns zero (0.0).
WAT?!!
 If str does not point to a valid C-string, or if the
 converted value would be out of the range of values
 representable by a double, it causes undefined behavior.
WAT WAT WAT WAT WAT WAT..??!! Seriously though, I should not be this judgemental. This is all hindsight - perhaps there was no NaN to represent non-number back when `atof` was designed or something.
Aug 15
prev sibling parent reply SealabJaster <sealabjaster gmail.com> writes:
On Saturday, 14 August 2021 at 06:27:54 UTC, evilrat wrote:
 ...
Honestly it gets even worse when there's also the allure of 5 minutes of people's attention for going "Hey, guys, I just made an awful build tool, give me praise!" that I'm ever so often to fall for. Like literally, every post I've made in this thread I have to hold myself back from posting a link to the code, because there's always that slight satisfaction of others looking at your work, even when there's no need for it.
Aug 15
parent reply SealabJaster <sealabjaster gmail.com> writes:
On Monday, 16 August 2021 at 01:14:22 UTC, SealabJaster wrote:
 ...
idk if anyone else _also_ has this problem, but does anyone else feel like they're a literal addict to coding, in a bad way? It's been really bad for me the last few weeks, where instead of studying, exercising, or doing anything that isn't coding, I'll just spend days tapping away at completely useless projects that don't see the light of day just to get my "fix" of problem solving/whatever it is my brain wants. And then you hit a wall where there's nothing you're working on that satisfies your brain's hunger. You're not working on anything hard, or useful, or meaningful, and so the feelings just fester and bring your down that you can't satisfy your urges to make things. It does sound ridiculous, but it's an honest issue I'm facing, and was just wondering if anyone else experiences this.
Aug 20
parent reply Abdulhaq <alynch4047 gmail.com> writes:
On Friday, 20 August 2021 at 16:43:10 UTC, SealabJaster wrote:
 On Monday, 16 August 2021 at 01:14:22 UTC, SealabJaster wrote:
 ...
idk if anyone else _also_ has this problem, but does anyone else feel like they're a literal addict to coding, in a bad way? It's been really bad for me the last few weeks, where instead of studying, exercising, or doing anything that isn't coding, I'll just spend days tapping away at completely useless projects that don't see the light of day just to get my "fix" of problem solving/whatever it is my brain wants. And then you hit a wall where there's nothing you're working on that satisfies your brain's hunger. You're not working on anything hard, or useful, or meaningful, and so the feelings just fester and bring your down that you can't satisfy your urges to make things. It does sound ridiculous, but it's an honest issue I'm facing, and was just wondering if anyone else experiences this.
I understand this, and from my own experience I've found that I like to distract myself with tasks like this when I'm trying to avoid thinking about or dealing with some unpleasantness in my life, which maybe I'm not even fully conscious of. The concentration of thinking takes away the danger of having to contemplate the unpleasant thing that we don't want to face or think about.
Aug 21
parent reply SealabJaster <sealabjaster gmail.com> writes:
On Saturday, 21 August 2021 at 09:02:58 UTC, Abdulhaq wrote:
 ...
This is probably it to be honest.
Aug 22
parent reply Abdulhaq <alynch4047 gmail.com> writes:
On Sunday, 22 August 2021 at 15:07:24 UTC, SealabJaster wrote:
 On Saturday, 21 August 2021 at 09:02:58 UTC, Abdulhaq wrote:
 ...
This is probably it to be honest.
We hackers are really fortunate that our distraction activity is so wholesome, we don't turn to booze or whatever other vastly more destructive escapes!
Aug 23
parent reply SealabJaster <sealabjaster gmail.com> writes:
On Monday, 23 August 2021 at 08:01:27 UTC, Abdulhaq wrote:
 ...
If only something actually useful ever came out of it though. I keep escalating the complexity of the things I want to do: "I'll write a build tool!" -> "I'll write a package manager!" -> "I'll write an assembler!" "Then a JIT, or an OS, then, then, then....." And nothing I actually write will be used or useful in any meaningful way. Though tbf even when I try to write something useful it still goes unused anyway.
Aug 24
parent Abdulhaq <alynch4047 gmail.com> writes:
On Tuesday, 24 August 2021 at 07:42:36 UTC, SealabJaster wrote:
 If only something actually useful ever came out of it though.

 I keep escalating the complexity of the things I want to do:

 "I'll write a build tool!"
 "I'll write a package manager!"

 "I'll write an assembler!"

 "Then a JIT, or an OS, then, then, then....."
I think many of us experience this TBH. Again I'm in the same position but as someone in his 50s I can say that the skills, craft and experience learnt during the process of building these tools comes in useful at certain key junctures in our later careers. Also, hopefully, the building and learning was fun, even if no-one else witnesses it! The explosion of social media, hacker news etc., has polluted out thought processes to the extent where we can end up wanting approval for what we have done, and feelings of failure if the work is not recognised. When this happens to me I reflect on what my life goals really are (and they don't include getting recognition for my pet projects) and that helps me orient where I place my efforts and the satisfaction derived from them (in theory, anyway :-) ). On a side note, I once watched an episode of Chasing Classic Cars and a recenty widowed lady was selling her husband's prized classic car. Passing through his workshop and referring to the lathes etc., she commented: He was building a tool to build a tool to build a tool. I totally get this!
Aug 29
prev sibling parent Paul Backus <snarwin gmail.com> writes:
On Tuesday, 10 August 2021 at 01:11:45 UTC, SealabJaster wrote:
 * It appears to be very difficult to make a build tool that 
 isn't just `make` with a pretty coat of paint?
I haven't actually tried it with a real project yet, but one build tool I find interesting in this regard is `redo`: https://redo.readthedocs.io/en/latest/ With `redo`, you specify your project's build process as a collection of *imperative* scripts, which generate dependency information *as a side effect*. This ensures that your dependency graph is always up-to-date, without requiring any of the [crazy hacks][1] you'd need for GNU `make`. [1]: http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/
Aug 13