www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - [Semi-OT] On the topic of build tools

reply SealabJaster <sealabjaster gmail.com> writes:
This is a rather aimless discussion, but I wanted to get this off 
my mind since I've been hopping around different build systems 
recently. Also it's 2am.

I've recently started to move out of my shell of cushy build 
systems managed by an IDE, or super simple ones like Dub, and 
have tried out a few different build tools now:

* CMake
* Scons
* Meson
* Bazel

Cmake: I like it when I'm not the one writing it. Its language 
feels like it has the same backwards compat and weird syntax 
issues that C++ deals with, which I found kind of ironic.

Scons: I don't know why but I couldn't get my head around it. I 
found it confusing where I simply wanted to include a simple D 
library into a simple D app.

Meson: I actually kind of like Meson because it seems to focus on 
being a build tool and not a complete language + dep manager + 
everything and the sink. It does feel a bit rigid, and the D 
support is iffy (I still can't figure out how to make the auto 
dub config spit out dub dependencies), but was probably the most 
useable one for D.

Bazel: I'm still mentally recovering.

My questions are simple, but I fear the answers are far, far from 
it:

* Are there any build systems are aren't... 'awful', 'anti-fun', 
etc?

* Build tools for some reason decide that they should also be 
pseudo or actual package managers. How much could a realistic 
tool be decoupled from having to manage dependencies (it'd of 
course need to know about subprojects and the like).

* What would the ideal build tool look like, and be able to do? 
Especially in regards to its configuration language.

* It appears to be very difficult to make a build tool that isn't 
just `make` with a pretty coat of paint?

Some tools feel so rigid and inflexible, even if you stay within 
bounds of its supported languages, whereas other tools feel far 
too 'free' in a sense that there's no clear direction on how to 
do certain things, even if they're technically possible.

As my autism-ridden brain loves to do, it got rather obsessed 
with this topic for a while, and I simply wanted to see what the 
general feelings on these tools are on this forum, because all 
the tools seem to be "necessary evils" rather than "gifts from 
god" in terms of their usage and popularity compared to their 
difficulties and downfalls.

I find Bazel's approach quite interesting: being able to download 
tools and language support straight from the same language that 
configures your builds. I believe language support is also added 
via said language.

I found the idea of using two languages interesting: e.g. use 
something like LUA to defines tooling, how to build X files, 
etc., then something more constrained and declarative like SDL to 
make use of the things the backend language provides. So 
essentially a seperation of concerns between the extension and 
configuration languages.

Sorry if this is a bit incoherent, but I'm finding it kind of 
hard to express what my thoughts are on the topic. I guess my 
main desire is to simply start a discussion on what potential a 
'proper' build tool could have compared to what's currently out 
there.

I've seen previous discussions before about separation between 
building, dependency management, dependency+tool fetching, etc. 
and it feels like an interesting topic for how 'bad', fractured, 
and so on the actual tooling is.

Also: https://xkcd.com/927/

Apologies for the faff, but I literally feel an urge to post this 
for some reason, and it's difficult to resist that feeling.

(also p.s., as I tend to do, I'll wake up tomorrow and feel too 
scared to reply, so I'll definitely see any replies, but may not 
respond. Sorry!)
Aug 09 2021
next sibling parent Tejas <notrealemail gmail.com> writes:
On Tuesday, 10 August 2021 at 01:11:45 UTC, SealabJaster wrote:

 This is a rather aimless discussion, but I wanted to get this 
 off my mind since I've been hopping around different build 
 systems recently. Also it's 2am.

 [...]


If someone us really going to read, please also help in 
determining whether Microsoft's vcpkg can also be used for 
building D projects.

Thanks in advance.
Aug 10 2021
prev sibling next sibling parent Dukc <ajieskola gmail.com> writes:
On Tuesday, 10 August 2021 at 01:11:45 UTC, SealabJaster wrote:

 * Are there any build systems are aren't... 'awful', 
 'anti-fun', etc?


Quickly reading, it seems that you might like 
[Reggae](https://code.dlang.org/packages/reggae)


 * Build tools for some reason decide that they should also be 
 pseudo or actual package managers. How much could a realistic 
 tool be decoupled from having to manage dependencies (it'd of 
 course need to know about subprojects and the like).


The ones that I know and do so (DUB and Nix) can both be 
configured to use local directories  as dependencies if you 
prefer to fetch stuff manually or with another package manager.
Aug 10 2021
prev sibling next sibling parent Denis Feklushkin <denis.feklushin gmail.com> writes:
On Tuesday, 10 August 2021 at 01:11:45 UTC, SealabJaster wrote:

 * Meson


Pros:

* It is hard to write working but unmaintainable scripts. 
Language constructed by way that Meson config files isn't evolve 
into spaghetti.
* Many popular projects now move to Meson
* Doesn't use (even if you really want to) the source tree for 
storing intermediate files or resulting binaries
* Language independent
* Easier to learn than cmake
* Active development, many commits merged each day

Cons:

* Harder to learn than DUB
* Only few mesonified D packages
* A lot of D-related bugs
* Still need to understanding what really goes when you compile 
something, this isn't silver bullet
Aug 10 2021
prev sibling next sibling parent reply Domain <dont_email empty.com> writes:
On Tuesday, 10 August 2021 at 01:11:45 UTC, SealabJaster wrote:

 This is a rather aimless discussion, but I wanted to get this 
 off my mind since I've been hopping around different build 
 systems recently. Also it's 2am.

 [...]


https://github.com/xmake-io/xmake

You can try xmake.
Aug 10 2021
parent evilrat <evilrat666 gmail.com> writes:
On Wednesday, 11 August 2021 at 00:02:38 UTC, Domain wrote:

 On Tuesday, 10 August 2021 at 01:11:45 UTC, SealabJaster wrote:

 This is a rather aimless discussion, but I wanted to get this 
 off my mind since I've been hopping around different build 
 systems recently. Also it's 2am.

 [...]


 https://github.com/xmake-io/xmake

 You can try xmake.


Looks nice and slim, but... I see too much stringly typed stuff 
using scripting language, not sure if it'll work for me over 
CMake even in mixed C++/D projects.

Also what catches my eyes is use of target() declaration and 
nested block, looks fishy to me.
Aug 12 2021
prev sibling next sibling parent reply ted <spam spam.com> writes:
On 10/8/21 10:41 am, SealabJaster wrote:

 This is a rather aimless discussion, but I wanted to get this off my 
 mind since I've been hopping around different build systems recently. 
 Also it's 2am.
 


Have a look at:

https://github.com/GrahamStJack/bottom-up-build

Used in significant projects with both C++ and D involved. Handles code 
generated during a build (& its dependencies). Fairly simple 
configuration/setup.

Enforces simple dependencies (will refuse to build if there are circular 
dependencies).
Aug 11 2021
parent SealabJaster <sealabjaster gmail.com> writes:
Several people wrote:

 ...


Thank you all for your suggestions, XMake especially seems quite 
interesting to try out.


 A lot of D-related bugs


_oh yes there are_, regardless though, I think Meson is currently 
my personal favourite.

For the last few days I've been toying with the idea of "what 
would my own build tool look like?" and even hacked together 
something.

But what I'm finding is: it's quite difficult to design something 
cohesive; simple by default yet powerful when opted-in, and just 
in general is useful enough to be worth using instead of already 
existing tools (who would've thought). My lack of experience 
certainly doesn't help either >x)

Maybe there's a valid reason 200 different tools exist...
Aug 11 2021
prev sibling next sibling parent reply ruki <waruqi gmail.com> writes:
On Tuesday, 10 August 2021 at 01:11:45 UTC, SealabJaster wrote:

 This is a rather aimless discussion, but I wanted to get this 
 off my mind since I've been hopping around different build 
 systems recently. Also it's 2am.

 [...]


xmake can build a mixed c/dlang project, and supports built-in 
package dependency management. It also supports downloading 
integrated dlang modules from dub. For example 
add_requires("dub::zlib")


https://github.com/xmake-io/xmake/blob/test/tests/projects/dlang/console_with_pkgs/xmake.lua
Aug 13 2021
parent reply SealabJaster <sealabjaster gmail.com> writes:
On Friday, 13 August 2021 at 14:08:29 UTC, ruki wrote:

 xmake can build a mixed c/dlang project, and supports built-in 
 package dependency management. It also supports downloading 
 integrated dlang modules from dub. For example 
 add_requires("dub::zlib")


Xmake is looking more and more interesting >:D


 imperative scripts, which generate dependency information as a 
 side effect


This was kind of what my hackish experiment was going for. You'd 
define things in LUA that manually called "isOutOfDate(file, 
{metadata})" and "update(file, {metadata})", and then the SDL 
half was just a list of function calls (quite literally since the 
SDL was translated into LUA).

Another pretty OT question, but does anyone else have this 
ridiculous urge to make their own versions of everything?

Like, instead of being sensible and using an already existing 
thing, you instead want to make a crappier, buggier, unfinished 
version simply because its "mine".
Aug 13 2021
next sibling parent Mike Parker <aldacron gmail.com> writes:
On Friday, 13 August 2021 at 19:30:29 UTC, SealabJaster wrote:


 This was kind of what my hackish experiment was going for. 
 You'd define things in LUA that manually called 
 "isOutOfDate(file, {metadata})" and "update(file, {metadata})", 
 and then the SDL half was just a list of function calls (quite 
 literally since the SDL was translated into LUA).


Premake uses Lua:

https://premake.github.io/

Manu got D support into it quite a while back. Not sure where it 
stands now.
Aug 13 2021
prev sibling parent reply evilrat <evilrat666 gmail.com> writes:
On Friday, 13 August 2021 at 19:30:29 UTC, SealabJaster wrote:

 Another pretty OT question, but does anyone else have this 
 ridiculous urge to make their own versions of everything?

 Like, instead of being sensible and using an already existing 
 thing, you instead want to make a crappier, buggier, unfinished 
 version simply because its "mine".


I do, a lot of people have that issue as well. This isn't really 
an issue until it is controlled, but when it's not the end result 
is almost empty GH page and a broken career.

Every time I need to start a project I have this:

   "meh, these libraries sucks, the author was stupid",

and it is easy to let it loose like:

   "ugh, the tooling is bloat and buggy, I'll make my own IDE 
quickly",

and then in some rare cases it evolves:

   "this OS is junk, if I am to move forward I will need my own 
OS".

So what is the deal with it?
   Well, when you rejected all libraries for the task (especially 
for reasons like style) it is time to stop, take a deep breath 
and swallow (your ego), remember - you have things to do, and if 
it is a paid job they pays you for that specific things and not 
for reinventing the wheel.
   Additionally you will have deadlines, and you had to support 
that thing afterwards for undetermined time. Would you really 
like supporting all your inventions for a lifetime? Probably not.
   And then if you just started to make that replacement for a 
specific library when you started to plan things and implementing 
it you will definitely hit some issues related to subject that 
simply wasn't there when you were planning, and this is where you 
probably start to understand why authors of those "stupid" 
libraries did "stupid" things.
   At this point it is time to stop doing crap, turn down your 
pride, thank the authors for saved time, and start using the said 
libraries.
   Enjoy your life with much less stress and depression!

Now, instead of "libraries" you could as well put "software" and 
it will be the same, you can even put "hardware" and it still 
will be the same.
And that is how it works!

Just remember - when you see "junk" or "bloat" it well could be 
the author's poor skills or incompetency (everyone hate amateurs, 
eh?), or it could be a real problem that was solved using what 
was really objectively possible at that moment because no one 
else did that before (or at least doesn't bothered to share, 
can't blame them though). And the latter is much more possible, 
unless of course it's just 300 lines of code prototype made at 
Friday's evening.
Aug 13 2021
next sibling parent Dukc <ajieskola gmail.com> writes:
On Saturday, 14 August 2021 at 06:27:54 UTC, evilrat wrote:

 I do, a lot of people have that issue as well. This isn't 
 really an issue until it is controlled, but when it's not the 
 end result is almost empty GH page and a broken career.


Might I add that there is a middle-road way to deal with this. 
One obviously should not develop everything from ground up, but I 
think it's okay to apply small personal patches to editors, 
version control or compilers, for example.

Also for simple stuff, like build scripts or syntax colouring 
schemes, it's totally reasonable to write your own if that works 
better.

Of course, you should always mind compatibility with others when 
doing this. If you tweak your DMD for example, I think it's 
reasonable to make tweaks to it's error reporting format, but not 
to the language semantics - otherwise you are not writing D 
anymore. The latter can be done if you're not expecting anybody 
else to work with your code.


 Every time I need to start a project I have this:

   "meh, these libraries sucks, the author was stupid",


Guess where I though kind-of like that? When reading the API of C 
standard library!

https://cplusplus.com/reference/cstdlib/atof/


 If no valid conversion could be performed, the function
 returns zero (0.0).


WAT?!!


 If str does not point to a valid C-string, or if the
 converted value would be out of the range of values
 representable by a double, it causes undefined behavior.


WAT WAT WAT WAT WAT WAT..??!!

Seriously though, I should not be this judgemental. This is all 
hindsight - perhaps there was no NaN to represent non-number back 
when `atof` was designed or something.
Aug 15 2021
prev sibling parent reply SealabJaster <sealabjaster gmail.com> writes:
On Saturday, 14 August 2021 at 06:27:54 UTC, evilrat wrote:

 ...


Honestly it gets even worse when there's also the allure of 5 
minutes of people's attention for going "Hey, guys, I just made 
an awful build tool, give me praise!" that I'm ever so often to 
fall for.

Like literally, every post I've made in this thread I have to 
hold myself back from posting a link to the code, because there's 
always that slight satisfaction of others looking at your work, 
even when there's no need for it.
Aug 15 2021
parent reply SealabJaster <sealabjaster gmail.com> writes:
On Monday, 16 August 2021 at 01:14:22 UTC, SealabJaster wrote:

 ...


idk if anyone else _also_ has this problem, but does anyone else 
feel like they're a literal addict to coding, in a bad way?

It's been really bad for me the last few weeks, where instead of 
studying, exercising, or doing anything that isn't coding, I'll 
just spend days tapping away at completely useless projects that 
don't see the light of day just to get my "fix" of problem 
solving/whatever it is my brain wants.

And then you hit a wall where there's nothing you're working on 
that satisfies your brain's hunger. You're not working on 
anything hard, or useful, or meaningful, and so the feelings just 
fester and bring your down that you can't satisfy your urges to 
make things.

It does sound ridiculous, but it's an honest issue I'm facing, 
and was just wondering if anyone else experiences this.
Aug 20 2021
parent reply Abdulhaq <alynch4047 gmail.com> writes:
On Friday, 20 August 2021 at 16:43:10 UTC, SealabJaster wrote:

 On Monday, 16 August 2021 at 01:14:22 UTC, SealabJaster wrote:

 ...


 idk if anyone else _also_ has this problem, but does anyone 
 else feel like they're a literal addict to coding, in a bad way?

 It's been really bad for me the last few weeks, where instead 
 of studying, exercising, or doing anything that isn't coding, 
 I'll just spend days tapping away at completely useless 
 projects that don't see the light of day just to get my "fix" 
 of problem solving/whatever it is my brain wants.

 And then you hit a wall where there's nothing you're working on 
 that satisfies your brain's hunger. You're not working on 
 anything hard, or useful, or meaningful, and so the feelings 
 just fester and bring your down that you can't satisfy your 
 urges to make things.

 It does sound ridiculous, but it's an honest issue I'm facing, 
 and was just wondering if anyone else experiences this.


I understand this, and from my own experience I've found that I 
like to distract myself with tasks like this when I'm trying to 
avoid thinking about or dealing with some unpleasantness in my 
life, which maybe I'm not even fully conscious of. The 
concentration of thinking takes away the danger of having to 
contemplate the unpleasant thing that we don't want to face or 
think about.
Aug 21 2021
parent reply SealabJaster <sealabjaster gmail.com> writes:
On Saturday, 21 August 2021 at 09:02:58 UTC, Abdulhaq wrote:

 ...


This is probably it to be honest.
Aug 22 2021
parent reply Abdulhaq <alynch4047 gmail.com> writes:
On Sunday, 22 August 2021 at 15:07:24 UTC, SealabJaster wrote:

 On Saturday, 21 August 2021 at 09:02:58 UTC, Abdulhaq wrote:

 ...


 This is probably it to be honest.


We hackers are really fortunate that our distraction activity is 
so wholesome, we don't turn to booze or whatever other vastly 
more destructive escapes!
Aug 23 2021
parent reply SealabJaster <sealabjaster gmail.com> writes:
On Monday, 23 August 2021 at 08:01:27 UTC, Abdulhaq wrote:

 ...


If only something actually useful ever came out of it though.

I keep escalating the complexity of the things I want to do:

"I'll write a build tool!"

->

"I'll write a package manager!"

->

"I'll write an assembler!"

"Then a JIT, or an OS, then, then, then....."

And nothing I actually write will be used or useful in any 
meaningful way.

Though tbf even when I try to write something useful it still 
goes unused anyway.
Aug 24 2021
next sibling parent Abdulhaq <alynch4047 gmail.com> writes:
On Tuesday, 24 August 2021 at 07:42:36 UTC, SealabJaster wrote:

 If only something actually useful ever came out of it though.

 I keep escalating the complexity of the things I want to do:

 "I'll write a build tool!"



 "I'll write a package manager!"

 "I'll write an assembler!"

 "Then a JIT, or an OS, then, then, then....."


I think many of us experience this TBH. Again I'm in the same 
position but as someone in his 50s I can say that the skills, 
craft and experience learnt during the process of building these 
tools comes in useful at certain key junctures in our later 
careers. Also, hopefully, the building and learning was fun, even 
if no-one else witnesses it! The explosion of social media, 
hacker news etc., has polluted out thought processes to the 
extent where we can end up wanting approval for what we have 
done, and feelings of failure if the work is not recognised. When 
this happens to me I reflect on what my life goals really are 
(and they don't include getting recognition for my pet projects) 
and that helps me orient where I place my efforts and the 
satisfaction derived from them (in theory, anyway :-) ).

On a side note, I once watched an episode of Chasing Classic Cars 
and a recenty widowed lady was selling her husband's prized 
classic car. Passing through his workshop and referring to the 
lathes etc., she commented: He was building a tool to build a 
tool to build a tool. I totally get this!
Aug 29 2021
prev sibling parent harakim <harakim gmail.com> writes:
On Tuesday, 24 August 2021 at 07:42:36 UTC, SealabJaster wrote:

 On Monday, 23 August 2021 at 08:01:27 UTC, Abdulhaq wrote:

 ...


 If only something actually useful ever came out of it though.

 I keep escalating the complexity of the things I want to do:

 "I'll write a build tool!"

 ->

 "I'll write a package manager!"

 ->

 "I'll write an assembler!"

 "Then a JIT, or an OS, then, then, then....."

 And nothing I actually write will be used or useful in any 
 meaningful way.

 Though tbf even when I try to write something useful it still 
 goes unused anyway.


I agree with the previous posters. I believe this is a normal 
phenomenon for good programmers, especially early in their 
careers. You're exploring the problem space and it WILL help you 
down the line.

I worked on a scheduler for the Cell Broadband Engine for about a 
year. I was not qualified at all to do it and I failed. It was 
too hard for me. There was no printing stuff out because that 
screwed up the scheduling. There was no debugging, period, 
outside of the emulators, which weren't a good enough emulation 
to rely on. Sometimes it would run for several minutes and then 
the result would be completely incorrect with no clues as to why. 
It caused me to UNDERSTAND what I was doing. It caused me to be 
able to run code in my head and think it different ways. I still 
use those skills today.

I do think when you get into modes of programming a lot, you are 
probably avoiding something else in your life. Make sure to take 
time to just not think about anything. Don't direct your brain, 
let it produce the thoughts. Some people call this meditation. It 
will help get you out of cycles of thinking and you'll be less 
anxious.

You have a part to play in this world. When you figure out what 
it is, then you can just move toward it. You'll be motivated to 
keep going even when it's tough. You'll feel like you're 
accomplishing things. You'll have direction. The world is like an 
orchestra and we all need you to play your part.

Until then, feel free to write lots of "useless" tools. It will 
help you in the end.
Oct 30 2021
prev sibling next sibling parent Paul Backus <snarwin gmail.com> writes:
On Tuesday, 10 August 2021 at 01:11:45 UTC, SealabJaster wrote:

 * It appears to be very difficult to make a build tool that 
 isn't just `make` with a pretty coat of paint?


I haven't actually tried it with a real project yet, but one 
build tool I find interesting in this regard is `redo`:

https://redo.readthedocs.io/en/latest/

With `redo`, you specify your project's build process as a 
collection of *imperative* scripts, which generate dependency 
information *as a side effect*. This ensures that your dependency 
graph is always up-to-date, without requiring any of the [crazy 
hacks][1] you'd need for GNU `make`.

[1]: 
http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/
Aug 13 2021
prev sibling parent Kagamin <spam here.lot> writes:
On Tuesday, 10 August 2021 at 01:11:45 UTC, SealabJaster wrote:

 I find Bazel's approach quite interesting: being able to 
 download tools and language support straight from the same 
 language that configures your builds. I believe language 
 support is also added via said language.

 I found the idea of using two languages interesting: e.g. use 
 something like LUA to defines tooling, how to build X files, 
 etc., then something more constrained and declarative like SDL 
 to make use of the things the backend language provides. So 
 essentially a seperation of concerns between the extension and 
 configuration languages.


msbuild uses similar ideas: it provides DSL, tasks, targets, 
items, and the actual build process rest is implemented in sdk of 
your choice, your project specifies sdk that builds it, and the 
said sdk provides build scripts that implement the build process 
and handle your project. Tasks are loaded from libraries and 
provide turing completeness that DSL can't provide.
Nov 02 2021