• Graham St Jack (48/48) Jun 26 2013 Bottom-up-build (bub) is a build system written in D which supports
• Rob T (10/10) Jun 26 2013 This build system seems to be very well suited for building
• eles (2/6) Jun 27 2013 CTRL-Z works for me. I think it expects input.
• eles (2/3) Jun 27 2013 Ignore it. It just suspends it.
• Marco Leise (7/11) Jun 27 2013 You might want to check how many programs you thought to have
• Graham St Jack (3/17) Jun 27 2013 I recently made a big change to the inter-thread communication code, and...
• Graham St Jack (5/24) Jun 30 2013 Fixed the problem, and also:
• eles (3/9) Jul 02 2013 Please add an "install:" target to the Makefile.
• Graham St Jack (2/15) Jul 02 2013 A better makefile is in progress, and it will have an install target.
• Dicebot (4/4) Jun 27 2013 Hm, bub.. Sounds like it should work with 'dub' nicely ;)
• Graham St Jack (4/9) Jun 27 2013 This isn't a build tool for everyone, but it really does make a big
• Rob T (15/18) Jun 27 2013 Well I'm noticing some interesting concepts, such as being able
• Graham St Jack (13/32) Jun 27 2013 That is correct. A side effect of assigning headers to libraries is that...
• Andrei Alexandrescu (3/5) Jun 27 2013 http://www.reddit.com/r/programming/comments/1h6p2w/bottomupbuild_a_buil...
• Marco Leise (7/7) Jun 27 2013 How does this build tool handle projects with multiple
• Graham St Jack (9/13) Jun 28 2013 A central theme in bottom-up-build is to produce any number of build
• Marco Leise (5/25) Jun 28 2013 Sounds good!
• David Bryant (27/29) Jun 28 2013 I've worked with Graham in the past and can attest to bub's coolness.
• John Colvin (3/7) Jun 28 2013 Any chance we could know what team this is? (Sorry if this is
• Graham St Jack (5/12) Jun 29 2013 It is the development team at my previous workplace. I haven't asked the...
• Clive Hobson (14/32) Jun 29 2013 I am still part of that team. We still use the early predecessor
• Kagamin (20/20) Jul 01 2013 Hooo, a self-contained build tool? That's cool.
• Graham St Jack (20/38) Jul 02 2013 Not completely arbitrary. Yous define (currently one-line) commands in
• Dicebot (3/9) Jul 02 2013 I think this should be properly solved by making compiler a

Graham St Jack <graham.stjack internode.on.net> writes:

Bottom-up-build (bub) is a build system written in D which supports 
building of large C/C++/D projects. It works fine on Linux, with a 
Windows port nearly completed. It should work on OS-X, but I haven't 
tested it there. 

Bub is hosted on https://github.com/GrahamStJack/bottom-up-build.


Some of bub's features that are useful on large projects are:

Built files are located outside the source directory, using a different 
build directory for (say) debug, release, profile, etc.

Very simple configuration files, making the build infrastructure easy to 
maintain.

Automatic deduction of which libraries to link with.

Automatic execution and evaluation of tests.

Enforcement of dependency control, with prevention of circularities 
between modules and directories.

Generated files are not scanned for imports/includes until after they are 
up to date. This is a real enabler for code generation.

Files in the build directory that should not be there are automatically 
deleted. It is surprising how often a left-over build artifact can make 
you think that something works, only to discover your mistake after a 
commit. This feature eliminates that problem.

The dependency graph is accurate, maximising opportunity for multiple 
build jobs and thus speeding up builds significantly.


An early precursor to bub was developed to use on a large C++ project 
that had complex dependencies and used a lot of code generation. Bub is a 
major rewrite designed to be more general-purpose.

The positive effect of the bub precursor on the project was very 
significant. Examples of positive consequences are:

Well-defined dependencies and elimination of circularities changed the 
design so that implementation and testing proceeded from the bottom up.

Paying attention to dependencies eliminated many unnecessary ones, 
resulting in a substantial increase in the reusability of code. This was 
instrumental in changing the way subsequent projects were designed, so 
that they took advantage of the large (and growing) body of reusable code.
The reusable code improved in design and quality with each project that 
used it.

Tests were compiled, linked and executed very early in the build - 
typically immediately after the code under test. This meant that 
regressions were usually detected within a few seconds of initiating a 
build. This was transformative to work rate, and willingness to make 
sweeping changes.

Doing a clean is hardly ever necessary. This is important because it 
dramatically reduces the total amount of time that builds take, which 
matters on a large project (especially C++).

Having a build system that works with both C++ and D meant that it was 
easy to "slip" some D code into the project. Initially as scripts, then 
as utilities, and so on. Having side-by-side comparisons of D against 
bash scripts and C++ modules had the effect of turning almost all the 
other team members into D advocates.

"Rob T" <alanb ucora.com> writes:

This build system seems to be very well suited for building 
complex large projects in a sensible way.

I successfully tested the example build on Debian linux. I will 
definitely explore this further using one of my own projects.

One issue I immediately ran into, is when I run bub incorrectly 
it hangs after writing the "bail" message to console. ctrl-c does 
not kill it, and I have to run a process kill commandto terminate.

Seems it gets stuck in doBailer() while(true) loop, but I only 
glanced at the source quickly before posting back here.

--rt

"eles" <eles eles.com> writes:

On Thursday, 27 June 2013 at 05:44:16 UTC, Rob T wrote:
 One issue I immediately ran into, is when I run bub incorrectly 
 it hangs after writing the "bail" message to console. ctrl-c 
 does not kill it, and I have to run a process kill commandto 
 terminate.

CTRL-Z works for me. I think it expects input.

"eles" <eles eles.com> writes:

On Thursday, 27 June 2013 at 07:32:32 UTC, eles wrote:
 CTRL-Z works for me. I think it expects input.

Ignore it. It just suspends it.

Marco Leise <Marco.Leise gmx.de> writes:

Am Thu, 27 Jun 2013 10:26:01 +0200
schrieb "eles" <eles eles.com>:

 On Thursday, 27 June 2013 at 07:32:32 UTC, eles wrote:
 CTRL-Z works for me. I think it expects input.

 
 Ignore it. It just suspends it.

You might want to check how many programs you thought to have
"killed" like this so far in your running session. It might
eat up your RAM.

-- 
Marco

Graham St Jack <graham.stjack internode.on.net> writes:

On Thu, 27 Jun 2013 07:44:07 +0200, Rob T wrote:

 This build system seems to be very well suited for building complex
 large projects in a sensible way.
 
 I successfully tested the example build on Debian linux. I will
 definitely explore this further using one of my own projects.
 
 One issue I immediately ran into, is when I run bub incorrectly it hangs
 after writing the "bail" message to console. ctrl-c does not kill it,
 and I have to run a process kill commandto terminate.
 
 Seems it gets stuck in doBailer() while(true) loop, but I only glanced
 at the source quickly before posting back here.
 
 --rt

I recently made a big change to the inter-thread communication code, and 
there are a few problems there still. I will look into it.

Graham St Jack <graham.stjack internode.on.net> writes:

On Thu, 27 Jun 2013 22:49:41 +0000, Graham St Jack wrote:

 On Thu, 27 Jun 2013 07:44:07 +0200, Rob T wrote:
 
 This build system seems to be very well suited for building complex
 large projects in a sensible way.
 
 I successfully tested the example build on Debian linux. I will
 definitely explore this further using one of my own projects.
 
 One issue I immediately ran into, is when I run bub incorrectly it
 hangs after writing the "bail" message to console. ctrl-c does not kill
 it, and I have to run a process kill commandto terminate.
 
 Seems it gets stuck in doBailer() while(true) loop, but I only glanced
 at the source quickly before posting back here.
 
 --rt

 
 I recently made a big change to the inter-thread communication code, and
 there are a few problems there still. I will look into it.

Fixed the problem, and also:

* split bub.d up into several smaller files,

* added a Makefile to bootstrap the building of bub, and

* added a bub.cfg and Bubfile as a trivial example of how to use bub.

"eles" <eles eles.com> writes:

On Monday, 1 July 2013 at 03:26:22 UTC, Graham St Jack wrote:
 On Thu, 27 Jun 2013 22:49:41 +0000, Graham St Jack wrote:
 Fixed the problem, and also:

 * split bub.d up into several smaller files,

 * added a Makefile to bootstrap the building of bub, and

 * added a bub.cfg and Bubfile as a trivial example of how to 
 use bub.

Please add an "install:" target to the Makefile.

Thank you.

Graham St Jack <graham.stjack internode.on.net> writes:

On Tue, 02 Jul 2013 14:07:16 +0200, eles wrote:

 On Monday, 1 July 2013 at 03:26:22 UTC, Graham St Jack wrote:
 On Thu, 27 Jun 2013 22:49:41 +0000, Graham St Jack wrote: Fixed the
 problem, and also:

 * split bub.d up into several smaller files,

 * added a Makefile to bootstrap the building of bub, and

 * added a bub.cfg and Bubfile as a trivial example of how to use bub.

 
 Please add an "install:" target to the Makefile.
 
 Thank you.

A better makefile is in progress, and it will have an install target.

"Dicebot" <public dicebot.lv> writes:

Hm, bub.. Sounds like it should work with 'dub' nicely ;)

Looks promising and I'd really love to see some build tool other 
then rdmd getting to the point it can be called standard. 
Makefile's sometimes are just too inconvenient.

Graham St Jack <graham.stjack internode.on.net> writes:

On Thu, 27 Jun 2013 12:24:27 +0200, Dicebot wrote:

 Hm, bub.. Sounds like it should work with 'dub' nicely ;)

Maybe so...

 
 Looks promising and I'd really love to see some build tool other then
 rdmd getting to the point it can be called standard. Makefile's
 sometimes are just too inconvenient.


This isn't a build tool for everyone, but it really does make a big 
difference on big projects.

"Rob T" <alanb ucora.com> writes:

On Thursday, 27 June 2013 at 23:03:40 UTC, Graham St Jack wrote:
 This isn't a build tool for everyone, but it really does make a 
 big
 difference on big projects.

Well I'm noticing some interesting concepts, such as being able 
to associate an include or import file with the library to link 
in, so that it gets done automatically simply by using the 
include/import file, great idea assuming I understood that 
correctly.

One thing you may want to consider, or maybe this is already 
possible, is add the ability to optionally specify something like 
*.d so that all .d files get included from a folder, rather than 
have to always individually specify them manually.

Also more concise documentation with clearer examples would be 
invaluable. I'm currently unsure if I need to restructure my 
existing project folders to fit bub or if bub can be configured 
to fit my existing projects.

--rt

Graham St Jack <graham.stjack internode.on.net> writes:

On Fri, 28 Jun 2013 08:05:08 +0200, Rob T wrote:

 On Thursday, 27 June 2013 at 23:03:40 UTC, Graham St Jack wrote:
 This isn't a build tool for everyone, but it really does make a big
 difference on big projects.

 
 Well I'm noticing some interesting concepts, such as being able to
 associate an include or import file with the library to link in, so that
 it gets done automatically simply by using the include/import file,
 great idea assuming I understood that correctly.

That is correct. A side effect of assigning headers to libraries is that 
bub can then enforce the visibility rules and give understandable error 
messages almost instantaneously when you transgress them. Another 
consequence is that the compiler only needs to be given a few -I 
directives because you aren't using them to control visibility.

 
 One thing you may want to consider, or maybe this is already possible,
 is add the ability to optionally specify something like *.d so that all
 .d files get included from a folder, rather than have to always
 individually specify them manually.

A useful idea. I will add it to the list.

 
 Also more concise documentation with clearer examples would be
 invaluable. I'm currently unsure if I need to restructure my existing
 project folders to fit bub or if bub can be configured to fit my
 existing projects.

Documentation is always tough to pitch at the right level. I will see 
what I can do.

Whether you need to restructure your project depends on whether you have 
already thought about dependencies and arranged things accordingly.

My suggestion is to extend the example a bit and get a feel for how it 
works, then think about what that means for your project.

Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:

On 6/26/13 5:10 PM, Graham St Jack wrote:
 Bottom-up-build (bub) is a build system written in D which supports
 building of large C/C++/D projects.

http://www.reddit.com/r/programming/comments/1h6p2w/bottomupbuild_a_build_system_for_ccd/


Andrei

Marco Leise <Marco.Leise gmx.de> writes:

How does this build tool handle projects with multiple
executables ? For example the util-linux package contains
dozens of utilities or a project might have a CLI and a GUI
version. Or there might be slight alterations like setting
a version or debug flag: -debug=threading -version=demo

-- 
Marco

Graham St Jack <graham.stjack internode.on.net> writes:

On Fri, 28 Jun 2013 08:28:03 +0200, Marco Leise wrote:

 How does this build tool handle projects with multiple executables ? For
 example the util-linux package contains dozens of utilities or a project
 might have a CLI and a GUI version. Or there might be slight alterations
 like setting a version or debug flag: -debug=threading -version=demo


A central theme in bottom-up-build is to produce any number of build 
artifacts: libraries, executables, tests, etc.

Different versions are catered for with:
  bub-config --mode=<whatever> <build-dir>

The mode is described in your configuration file, and results in a build 
directory set up specifically for that mode.

If you want multiple modes, use multiple build directories.

Take a look in example/bub.cfg.

Marco Leise <Marco.Leise gmx.de> writes:

Am Fri, 28 Jun 2013 07:03:27 +0000 (UTC)
schrieb Graham St Jack <graham.stjack internode.on.net>:

 On Fri, 28 Jun 2013 08:28:03 +0200, Marco Leise wrote:
 
 How does this build tool handle projects with multiple executables ? For
 example the util-linux package contains dozens of utilities or a project
 might have a CLI and a GUI version. Or there might be slight alterations
 like setting a version or debug flag: -debug=threading -version=demo

 
 
 A central theme in bottom-up-build is to produce any number of build 
 artifacts: libraries, executables, tests, etc.
 
 Different versions are catered for with:
   bub-config --mode=<whatever> <build-dir>
 
 The mode is described in your configuration file, and results in a build 
 directory set up specifically for that mode.
 
 If you want multiple modes, use multiple build directories.
 
 Take a look in example/bub.cfg.

Sounds good!

-- 
Marco

David Bryant <bagnose gmail.com> writes:

On 27/06/13 09:40, Graham St Jack wrote:
 Bottom-up-build (bub) is a build system written in D which supports
 building of large C/C++/D projects.

I've worked with Graham in the past and can attest to bub's coolness.

My pet project 'terminol' is about a month away from an initial release, 
but given that is uses the bub build system, and that Graham has made 
his announcement, it could be interest now to anyone wishing to see a 
small project that uses bub:

   https://github.com/bagnose/terminol

Briefly: terminol is a lightweight terminal emulator for Linux, 
essentially aimed at being a rxvt-unicode refresh. It should checkout, 
build fine with bub, but so far I'm the only one who ever has. I'm 
sorry, it's not written in D (it's written in C++11) but I am a huge fan 
of D.

Having worked with GNU Make, Jam and autotools (and looked closely at 
several others) I find bub very pleasant to use indeed. I won't repeat 
the benefits that Graham already espoused.

I think the focus with bub thus far has been to develop its core 
approaches: unifying C/C++/D, bottom-up-build, library inference, 
circularity detection, etc. As such, it is a little unfurnished at the 
edges. For example, bub does not have a defined scheme for determining 
things like: what libraries does the user have available and what are 
their peculiarities (versions, location, etc). At the moment the user 
may need to do a bit of bub.cfg customisation of their own to make 
things work.

Those things are the next hurdle and will be interesting to discuss. 
However I think bub already demonstrates the promise of greatness and is 
worth taking a good look at.

David Bryant

"John Colvin" <john.loughran.colvin gmail.com> writes:

On Thursday, 27 June 2013 at 00:10:37 UTC, Graham St Jack wrote:
 Having side-by-side comparisons of D against
 bash scripts and C++ modules had the effect of turning almost 
 all the
 other team members into D advocates.

Any chance we could know what team this is? (Sorry if this is 
common knowledge)

Graham St Jack <graham.stjack internode.on.net> writes:

On Sat, 29 Jun 2013 00:59:15 +0200, John Colvin wrote:

 On Thursday, 27 June 2013 at 00:10:37 UTC, Graham St Jack wrote:
 Having side-by-side comparisons of D against bash scripts and C++
 modules had the effect of turning almost all the other team members
 into D advocates.

 
 Any chance we could know what team this is? (Sorry if this is common
 knowledge)

It is the development team at my previous workplace. I haven't asked them 
for permission, so I would rather not say who they are.

David Bryant (the previous poster) was a member of that team though, and 
he will be happy to provide details.

"Clive Hobson" <clivetown-dev03 yahoo.com.au> writes:

On Saturday, 29 June 2013 at 07:54:30 UTC, Graham St Jack wrote:
 On Sat, 29 Jun 2013 00:59:15 +0200, John Colvin wrote:

 On Thursday, 27 June 2013 at 00:10:37 UTC, Graham St Jack 
 wrote:
 Having side-by-side comparisons of D against bash scripts and 
 C++
 modules had the effect of turning almost all the other team 
 members
 into D advocates.

 
 Any chance we could know what team this is? (Sorry if this is 
 common
 knowledge)

 It is the development team at my previous workplace. I haven't 
 asked them
 for permission, so I would rather not say who they are.

 David Bryant (the previous poster) was a member of that team 
 though, and
 he will be happy to provide details.

I am still part of that team.  We still use the early predecessor 
of bub to build several projects from a large C++ codebase, but I 
am looking forward to switching to bub at the end of the current 
project.

For example, one project uses ~2000 cc/h files containing 
~390,000 lines (before any code generation).  After a successfull 
build, running the build again takes about 2s to figure out that 
everything is up to date.  Knowing both build tools, I expect 
that time to remain about the same after the switch to bub.

Yes, I have become a D advocate.  Our production code is still 
mostly C++, but a large part of our system-level testing is now 
written in D (~15,000 lines).  Bash scripts are becoming rarer 
too. :)

"Kagamin" <spam here.lot> writes:

Hooo, a self-contained build tool? That's cool.
1. Are arbitrary make-style commands supported? For example, on 
windows one may want to compile resources. Resources consist of a 
declaration .rc file, icons and manifest files, which are 
compiled into .res file, though only .rc file is passed to the 
resource compiler, but other files are still dependencies of 
course.
2. How to do heterogeneous linking? Again on windows an 
executable can be linked from code .obj's, resource .res and 
module definition .def:
gcc main.o r.res mytool.def -o mytool.exe
3. Probably not actual for big projects. Can build workspase be 
created automatically by bub instead of explicitly by 
bub-config... for some default mode?
4. If a build server does builds from scratch, shouldn't it be 
better for performance to compile several source files in one 
command? Also applies to user builds: when one wants to install a 
project from source, he usually does it only once and from 
scratch. Have you any idea, how this affects compilation speed of 
c, c++ and d? Though dmd frontend can go out of memory.

Graham St Jack <graham.stjack internode.on.net> writes:

On Mon, 01 Jul 2013 18:32:54 +0200, Kagamin wrote:

 Hooo, a self-contained build tool? That's cool.

I like to think so.

 1. Are arbitrary make-style commands supported? For example, on windows
 one may want to compile resources. Resources consist of a declaration
 .rc file, icons and manifest files, which are compiled into .res file,
 though only .rc file is passed to the resource compiler, but other files
 are still dependencies of course.

Not completely arbitrary. Yous define (currently one-line) commands in 
bub.cfg, with access to variables that are expanded at the last moment 
before the command is issued. Some of the variables are pre-defined and 
populated by bub, and others you define yourself.

 2. How to do heterogeneous linking? Again on windows an executable can
 be linked from code .obj's, resource .res and module definition .def:
 gcc main.o r.res mytool.def -o mytool.exe

There isn't any fundamental reason why things like resource files can't 
be linked in, but I would probably have to make a few improvements. The 
use case that bub was forged in was linux-hosted C/C++ with lots of code 
generated from IDL files and the like.

 3. Probably not actual for big
 projects. Can build workspase be created automatically by bub instead of
 explicitly by bub-config... for some default mode?

You set up a build directory with bub-config. I can't currently see any 
reason why it would be useful for bub to invoke bub-config itself.

Re big projects - I guess it depends on what you mean by big. See the 
post from Clive Hobson, which describes a project that bub works well on. 
That project seems pretty big to me.


 4. If a build server does builds from scratch, shouldn't it be better
 for performance to compile several source files in one command? Also
 applies to user builds: when one wants to install a project from source,
 he usually does it only once and from scratch. Have you any idea, how
 this affects compilation speed of c, c++ and d? Though dmd frontend can
 go out of memory.

I assume you mean my preference for explicitly building object files. I 
like this because:
* It opens up more opportunities for using multiple cores (-j9)
* Dependency/reference tracking is simplified.
* Test utilities are easier if the code under test is available in a lib.

"Dicebot" <public dicebot.lv> writes:

On Monday, 1 July 2013 at 16:32:55 UTC, Kagamin wrote:
 4. If a build server does builds from scratch, shouldn't it be 
 better for performance to compile several source files in one 
 command? Also applies to user builds: when one wants to install 
 a project from source, he usually does it only once and from 
 scratch. Have you any idea, how this affects compilation speed 
 of c, c++ and d? Though dmd frontend can go out of memory.

I think this should be properly solved by making compiler a 
daemon/server, not by a build tool.