www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Redub: A faster build system promising 90% compatibility with dub

reply Hipreme <msnmancini hotmail.com> writes:
Hello, people.
Here I come announce a package I've just released:

https://code.dlang.org/packages/redub) : A faster 
build system based on dub

It is meant to be an almost drop-in replacement on dub.
Right now, I've been implementing only features that dub already 
has. But in the future, I can't guarantee I'll keep following the 
same direction from dub since there is a bunch I disagree with.



- This is not a *reggae* clone (it is a lot easier to use by the 
way). This is a dub that simply can build with parallelization.
- It completely ignores optional dependencies
- Version range won't be implemented, at least not by me



It has a very raw package management (by using dub CLI) and 
semver matching.



1. People with a project with many dependencies: Those people 
will get a big speedup on recompilations since each library can 
be built independently (if they don't mess up with pre/post 
commands)
2. People which don't rely on version ranges: Personally, I have 
0 interest on implementing that, if you wish support for it, 
issue a pull request.
3. People which wants rule-based software instead of exception 
based: The features are designed around rules instead of 
workarounds. If you need an exception in using it, I won't listen 
to this request or pull request. Only generalizations may be 
implemented.
4. People which wants better caching: Dub currently is completely 
buggy on caches with string import paths, (and actually, for some 
unknown reason, can't even detect up to date redub when running 
it. )



- dub recipe files are parsed lazily, which means, you can use 
inexistent dependencies inside other configurations, and also 
faster parsing even though I'm using std.json
- The code is following a 95% functional style. Almost no state 
is changed in between compilation steps
- This project is focused on readability and easy of maintenance: 
Don't try to be too smart, unless real difference can be made on 
its performance
- Libraries are always built recursively. (Currently was only 
achieved with --deep flag). Which for me was simply a big "why"
- The default output now is inside a separate folder: "bin". This 
makes it easier for making the caching system to behave better 
since they will be check on other root.
- I won't focus on any moment into implementing support for a 
single file build, this is too much work for no real gain, if you 
want to do it, do it on a completely isolated file.





- This software was built in 2.5 weeks, which first was a 1 week 
project, but, it was so successful in what it done that I decided 
to expand its scope (I would initially only support what Hipreme 
Engine needs). Some people said that no one done better than dub 
until now, so, I've done this project because it was simply 
unbelievable that dub wasn't able to do any level of 
parallelization.

- I have tried contributing to dub's project on parallelization, 
I waste 1 week trying that and could not get it working, so, I 
decided it would be a better use of my time into rewriting a 
completely new dub which would focus on being easy to read -- An 
example of that is how easily someone was able to integrate an 
experimental support to building C with it.


This project has been tested with
- [Hipreme Engine](https://code.dlang.org/packages/hipreme_engine)
- [Fluid](https://code.dlang.org/packages/fluid)
- [arsd](https://code.dlang.org/packages/arsd-official)
- [dplug](https://code.dlang.org/packages/dplug)


And possibly more to come. I didn't done a complete vibe-d 
support because there was too many features which I am not going 
to use (hello arch-based filters on dub.json).



- A full rebuild on Hipreme Engine reduced its compilation time 
from 14 seconds to 3. One could say that full rebuilds aren't 
common, but this is untrue. Specially when I was developing my 
own standard library (util), this would get into my nerves.

You may test it with `dub run redub`. But since `dub` is buggy, 
it will always be identified as not up to date :)


By the way: People which hates `dub` and use `dmd -i` instead: 
your project ought to be built faster by modularizing them :)
Jan 20
parent reply Renato <renato athaydes.com> writes:
On Saturday, 20 January 2024 at 14:14:10 UTC, Hipreme wrote:
 - I have tried contributing to dub's project on 
 parallelization, I waste 1 week trying that and could not get 
 it working, so, I decided it would be a better use of my time 
 into rewriting a completely new dub which would focus on being 
 easy to read -- An example of that is how easily someone was 
 able to integrate an experimental support to building C with it.
Could you expand on that? Perhaps someone else could continue your work. While I understand you preferred to write your own system (we all probably prefer greenfield development, after all that means it's fully yours, and you can fully understand it, own it, and change it as you see fit which is a very attractive proposition), I'm afraid your project will only be alive while you have interest in maintaining it (I've seen similar projects in other languages before, often they have one or two releases before the author moves on to the next greenfield project) - unless you manage to create an active community around it, which is exceedingly difficult and might be more than you're signing up for once a lot of people join in and start demanding features - while dub, even if it's so "buggy" as you claim, is the official tool everyone is likely to continue to use, and consequently improvements to it would be highly welcome.
Jan 20
next sibling parent Hipreme <msnmancini hotmail.com> writes:
On Saturday, 20 January 2024 at 15:25:35 UTC, Renato wrote:
 On Saturday, 20 January 2024 at 14:14:10 UTC, Hipreme wrote:
 - I have tried contributing to dub's project on 
 parallelization, I waste 1 week trying that and could not get 
 it working, so, I decided it would be a better use of my time 
 into rewriting a completely new dub which would focus on being 
 easy to read -- An example of that is how easily someone was 
 able to integrate an experimental support to building C with 
 it.
Could you expand on that? Perhaps someone else could continue your work. While I understand you preferred to write your own system (we all probably prefer greenfield development, after all that means it's fully yours, and you can fully understand it, own it, and change it as you see fit which is a very attractive proposition), I'm afraid your project will only be alive while you have interest in maintaining it (I've seen similar projects in other languages before, often they have one or two releases before the author moves on to the next greenfield project) - unless you manage to create an active community around it, which is exceedingly difficult and might be more than you're signing up for once a lot of people join in and start demanding features - while dub, even if it's so "buggy" as you claim, is the official tool everyone is likely to continue to use, and consequently improvements to it would be highly welcome.
I don't care if people actually use or not. Dub is not easy to understand, not easy to maintain, it is less performant than redub and the main reason for redub even exist was because my project (Hipreme Engine) needed a better solution. Reggae is also an official tool which no one uses. RDMD is an official solution that should be deprecated since the existence of RUND. Also, contributing to an official project would only slow down my progress, which for me, is not negotiable. There are a bunch of things which I've done because D doesn't move at the same speed as me, such as: - My WASM runtime - Make the runtime generic enough to support an unrelated platform (PSVita) - My solution to cross compilation which is able to automatically identify what is missing on an user setup for being able to release to non standard D platforms. - redub (the slow release of the flags `--deep` and `--recipe` made me stall for quite a lot more of time than I expected), the fully stateful and confuse code only made a lot of behaviors in dub that does not scale well. - My own standard library which does not depend on C runtime, which makes it a lot more portable since it becomes D-only code. - The existence of `objc_meta` which gave an easy and performant solution for calling Objective-C functions. Dub needs a rewrite, but it can't change, because they are caring about breaking dub as a library, even though versioning exists only for that. I have a lot to work right now and can't give more time to things that aren't an issue for me. Nonetheless, if this project suits you well, there's no reason not to use it.
Jan 20
prev sibling parent reply ryuukk_ <ryuukk.dev gmail.com> writes:
On Saturday, 20 January 2024 at 15:25:35 UTC, Renato wrote:
 On Saturday, 20 January 2024 at 14:14:10 UTC, Hipreme wrote:
 - I have tried contributing to dub's project on 
 parallelization, I waste 1 week trying that and could not get 
 it working, so, I decided it would be a better use of my time 
 into rewriting a completely new dub which would focus on being 
 easy to read -- An example of that is how easily someone was 
 able to integrate an experimental support to building C with 
 it.
Could you expand on that? Perhaps someone else could continue your work. While I understand you preferred to write your own system (we all probably prefer greenfield development, after all that means it's fully yours, and you can fully understand it, own it, and change it as you see fit which is a very attractive proposition), I'm afraid your project will only be alive while you have interest in maintaining it (I've seen similar projects in other languages before, often they have one or two releases before the author moves on to the next greenfield project) - unless you manage to create an active community around it, which is exceedingly difficult and might be more than you're signing up for once a lot of people join in and start demanding features - while dub, even if it's so "buggy" as you claim, is the official tool everyone is likely to continue to use, and consequently improvements to it would be highly welcome.
This quote says it all:
 A full rebuild on Hipreme Engine reduced its compilation time 
 from 14 seconds to 3
dub is over engineered We should encourage more of these projects, they defy the stalled status quo in D
Jan 20
parent reply Hipreme <msnmancini hotmail.com> writes:
On Saturday, 20 January 2024 at 16:03:52 UTC, ryuukk_ wrote:
 On Saturday, 20 January 2024 at 15:25:35 UTC, Renato wrote:
 On Saturday, 20 January 2024 at 14:14:10 UTC, Hipreme wrote:
 - I have tried contributing to dub's project on 
 parallelization, I waste 1 week trying that and could not get 
 it working, so, I decided it would be a better use of my time 
 into rewriting a completely new dub which would focus on 
 being easy to read -- An example of that is how easily 
 someone was able to integrate an experimental support to 
 building C with it.
Could you expand on that? Perhaps someone else could continue your work. While I understand you preferred to write your own system (we all probably prefer greenfield development, after all that means it's fully yours, and you can fully understand it, own it, and change it as you see fit which is a very attractive proposition), I'm afraid your project will only be alive while you have interest in maintaining it (I've seen similar projects in other languages before, often they have one or two releases before the author moves on to the next greenfield project) - unless you manage to create an active community around it, which is exceedingly difficult and might be more than you're signing up for once a lot of people join in and start demanding features - while dub, even if it's so "buggy" as you claim, is the official tool everyone is likely to continue to use, and consequently improvements to it would be highly welcome.
This quote says it all:
 A full rebuild on Hipreme Engine reduced its compilation time 
 from 14 seconds to 3
dub is over engineered We should encourage more of these projects, they defy the stalled status quo in D
Just came to announce that I've finished implementing redub as a library, present on version 1.2.1: ```d import redub.api; import redub.logging; void main() { import std.file; setLogLevel(LogLevel.verbose); ProjectDetails d = resolveDependencies( invalidateCache: false, std.system.os, CompilationDetails("dmd", "arch not yet implemented", "dmd v[2.105.0]"), ProjectToParse("configuration", getcwd(), "subPackage", "path/to/dub/recipe.json (optional)") ); buildProject(d); } ``` Beyond that, I've also done 2 new optimizations: 1. Fixed a problem where memory was being allocated in a non essential task, causing some GC pauses while resolving dependencies. This reduced dependency resolution time to 1/4. Hipreme Engine went from 200ms to 50m, now the bottleneck being mostly `std.json`, but I don't think it is worth changing it. 2. Added support for compiler version assumption. This makes it even faster since it won't need to keep rerunning the `compiler --version` each time. This is a minor optimization that got 30 ms faster (on Windows).
Jan 22
parent reply Hipreme <msnmancini hotmail.com> writes:
Hello guys. It has been some time since my last post on forum 
regarding redub.
It has improved a lot on its newest versions. Since it has been 
put to prove on Hipreme Engine as its main build system, there 
were a bunch of updates that I've done that made it even faster, 
reliable, with more support to features that one can encounter in 
dub. There is still a plenty of niche features that dub has that 
it wasn't implemented, but, by now, I'm pretty sure it should 
have achieved its 90% of compatibility :)

Here follows the release notes since the last time I posted



- Added: Support for folders with whitespace

v1.3.8
- Added: Error API
- Added: Public API for not requiring describe for getting linker 
files and source files

v1.3.7
- Added: --version and fixed private dependencies
- Fixed: Now hipjson parses a single value json

v1.3.4
- Fixed: Memory allocation was failing
- Fixed: Even faster JSON parsing

v1.3.3
- Update: Make JSON parsing of redub faster by using hipjson ( I 
got to parse json in 1/5 of the time)
- Fixed: Always use -op/--oq since it creates a compilation with 
better compatibility
v1.3.1
- Added: Support for repository based dub paths
- Added: buildType support

v1.2.20
- Added: Now rebuild is also based on output artifact. This 
guarantees that it won't reuse compilation artifacts on different 
configurations
- Fixed: Generate cache for static libs too

v1.2.18
- Update: Content hashing is now using a faster hashing algorithm

v1.2.17
- Update: Improved efficiency of cache generation
- Update: Improved cache generation
- Fixed: now adv_diff supports date-only comparison
- Fixed: Some bugs on serialization
- Fixed: Bug when some caches would not be correctly identified

v1.2.13
- Fixed: Redub describe now includes requirements extra libpaths
- Added: Support to dflags as link flags and fixed escaping 
problem
- Fixed: Support for private visibility of dependencies, 
outputting file names correctly based on target system, linker 
error are now being checked earlier
- Refactor: Buildapi is being prepared to use private visibility
- Added: Better cross compilation support, now also forwarding 
linking on msvc to default one when cross compiling. Command 
filters will now work on the target platform
- Added: Wasm support
- Fixed: Subconfiguration only are not iterated
- Fixed: Always merging sourcePaths into importPaths. finish 
implementation now only will traverse unvisited nodes
- Fixed: Now elements with subConfigurations which aren't default 
are the first to be parsed
- Fixed: Using Breadth First Search and mixing subConfigurations 
for better subConfiguration resolution
- Added: Architecture support
- Added: More documentation to redub
- Fixed: Now, when rebuilding, redub will remove existing 
binaries, since this was causing some problems with existing 
binaries
- Added: D Compiler abstraction for command generator. Also using 
more performatic versions of map (faster dependency resolution)
- Fixed: Redub environment support
- Added: Dub describe support, fixed problem on environment 
variables
May 04
parent aberba <karabutaworld gmail.com> writes:
On Saturday, 4 May 2024 at 11:40:27 UTC, Hipreme wrote:
 Hello guys. It has been some time since my last post on forum 
 regarding redub.
 It has improved a lot on its newest versions. Since it has been 
 put to prove on Hipreme Engine as its main build system, there 
 were a bunch of updates that I've done that made it even 
 faster, reliable, with more support to features that one can 
 encounter in dub. There is still a plenty of niche features 
 that dub has that it wasn't implemented, but, by now, I'm 
 pretty sure it should have achieved its 90% of compatibility :)

 [...]
Gonna try it soon.
May 06