digitalmars.D.announce - Announcing bottom-up-build - a build system for C/C++/D
- Graham St Jack (48/48) Jun 26 2013 Bottom-up-build (bub) is a build system written in D which supports
- Rob T (10/10) Jun 26 2013 This build system seems to be very well suited for building
- eles (2/6) Jun 27 2013 CTRL-Z works for me. I think it expects input.
- eles (2/3) Jun 27 2013 Ignore it. It just suspends it.
- Marco Leise (7/11) Jun 27 2013 You might want to check how many programs you thought to have
- Graham St Jack (3/17) Jun 27 2013 I recently made a big change to the inter-thread communication code, and...
- Graham St Jack (5/24) Jun 30 2013 Fixed the problem, and also:
- eles (3/9) Jul 02 2013 Please add an "install:" target to the Makefile.
- Graham St Jack (2/15) Jul 02 2013 A better makefile is in progress, and it will have an install target.
- Dicebot (4/4) Jun 27 2013 Hm, bub.. Sounds like it should work with 'dub' nicely ;)
- Graham St Jack (4/9) Jun 27 2013 This isn't a build tool for everyone, but it really does make a big
- Rob T (15/18) Jun 27 2013 Well I'm noticing some interesting concepts, such as being able
- Graham St Jack (13/32) Jun 27 2013 That is correct. A side effect of assigning headers to libraries is that...
- Andrei Alexandrescu (3/5) Jun 27 2013 http://www.reddit.com/r/programming/comments/1h6p2w/bottomupbuild_a_buil...
- Marco Leise (7/7) Jun 27 2013 How does this build tool handle projects with multiple
- Graham St Jack (9/13) Jun 28 2013 A central theme in bottom-up-build is to produce any number of build
- Marco Leise (5/25) Jun 28 2013 Sounds good!
- David Bryant (27/29) Jun 28 2013 I've worked with Graham in the past and can attest to bub's coolness.
- John Colvin (3/7) Jun 28 2013 Any chance we could know what team this is? (Sorry if this is
- Graham St Jack (5/12) Jun 29 2013 It is the development team at my previous workplace. I haven't asked the...
- Clive Hobson (14/32) Jun 29 2013 I am still part of that team. We still use the early predecessor
- Kagamin (20/20) Jul 01 2013 Hooo, a self-contained build tool? That's cool.
- Graham St Jack (20/38) Jul 02 2013 Not completely arbitrary. Yous define (currently one-line) commands in
- Dicebot (3/9) Jul 02 2013 I think this should be properly solved by making compiler a
Bottom-up-build (bub) is a build system written in D which supports building of large C/C++/D projects. It works fine on Linux, with a Windows port nearly completed. It should work on OS-X, but I haven't tested it there. Bub is hosted on https://github.com/GrahamStJack/bottom-up-build. Some of bub's features that are useful on large projects are: Built files are located outside the source directory, using a different build directory for (say) debug, release, profile, etc. Very simple configuration files, making the build infrastructure easy to maintain. Automatic deduction of which libraries to link with. Automatic execution and evaluation of tests. Enforcement of dependency control, with prevention of circularities between modules and directories. Generated files are not scanned for imports/includes until after they are up to date. This is a real enabler for code generation. Files in the build directory that should not be there are automatically deleted. It is surprising how often a left-over build artifact can make you think that something works, only to discover your mistake after a commit. This feature eliminates that problem. The dependency graph is accurate, maximising opportunity for multiple build jobs and thus speeding up builds significantly. An early precursor to bub was developed to use on a large C++ project that had complex dependencies and used a lot of code generation. Bub is a major rewrite designed to be more general-purpose. The positive effect of the bub precursor on the project was very significant. Examples of positive consequences are: Well-defined dependencies and elimination of circularities changed the design so that implementation and testing proceeded from the bottom up. Paying attention to dependencies eliminated many unnecessary ones, resulting in a substantial increase in the reusability of code. This was instrumental in changing the way subsequent projects were designed, so that they took advantage of the large (and growing) body of reusable code. The reusable code improved in design and quality with each project that used it. Tests were compiled, linked and executed very early in the build - typically immediately after the code under test. This meant that regressions were usually detected within a few seconds of initiating a build. This was transformative to work rate, and willingness to make sweeping changes. Doing a clean is hardly ever necessary. This is important because it dramatically reduces the total amount of time that builds take, which matters on a large project (especially C++). Having a build system that works with both C++ and D meant that it was easy to "slip" some D code into the project. Initially as scripts, then as utilities, and so on. Having side-by-side comparisons of D against bash scripts and C++ modules had the effect of turning almost all the other team members into D advocates.
Jun 26 2013
This build system seems to be very well suited for building complex large projects in a sensible way. I successfully tested the example build on Debian linux. I will definitely explore this further using one of my own projects. One issue I immediately ran into, is when I run bub incorrectly it hangs after writing the "bail" message to console. ctrl-c does not kill it, and I have to run a process kill commandto terminate. Seems it gets stuck in doBailer() while(true) loop, but I only glanced at the source quickly before posting back here. --rt
Jun 26 2013
On Thursday, 27 June 2013 at 05:44:16 UTC, Rob T wrote:One issue I immediately ran into, is when I run bub incorrectly it hangs after writing the "bail" message to console. ctrl-c does not kill it, and I have to run a process kill commandto terminate.CTRL-Z works for me. I think it expects input.
Jun 27 2013
On Thursday, 27 June 2013 at 07:32:32 UTC, eles wrote:CTRL-Z works for me. I think it expects input.Ignore it. It just suspends it.
Jun 27 2013
Am Thu, 27 Jun 2013 10:26:01 +0200 schrieb "eles" <eles eles.com>:On Thursday, 27 June 2013 at 07:32:32 UTC, eles wrote:You might want to check how many programs you thought to have "killed" like this so far in your running session. It might eat up your RAM. -- MarcoCTRL-Z works for me. I think it expects input.Ignore it. It just suspends it.
Jun 27 2013
On Thu, 27 Jun 2013 07:44:07 +0200, Rob T wrote:This build system seems to be very well suited for building complex large projects in a sensible way. I successfully tested the example build on Debian linux. I will definitely explore this further using one of my own projects. One issue I immediately ran into, is when I run bub incorrectly it hangs after writing the "bail" message to console. ctrl-c does not kill it, and I have to run a process kill commandto terminate. Seems it gets stuck in doBailer() while(true) loop, but I only glanced at the source quickly before posting back here. --rtI recently made a big change to the inter-thread communication code, and there are a few problems there still. I will look into it.
Jun 27 2013
On Thu, 27 Jun 2013 22:49:41 +0000, Graham St Jack wrote:On Thu, 27 Jun 2013 07:44:07 +0200, Rob T wrote:Fixed the problem, and also: * split bub.d up into several smaller files, * added a Makefile to bootstrap the building of bub, and * added a bub.cfg and Bubfile as a trivial example of how to use bub.This build system seems to be very well suited for building complex large projects in a sensible way. I successfully tested the example build on Debian linux. I will definitely explore this further using one of my own projects. One issue I immediately ran into, is when I run bub incorrectly it hangs after writing the "bail" message to console. ctrl-c does not kill it, and I have to run a process kill commandto terminate. Seems it gets stuck in doBailer() while(true) loop, but I only glanced at the source quickly before posting back here. --rtI recently made a big change to the inter-thread communication code, and there are a few problems there still. I will look into it.
Jun 30 2013
On Monday, 1 July 2013 at 03:26:22 UTC, Graham St Jack wrote:On Thu, 27 Jun 2013 22:49:41 +0000, Graham St Jack wrote: Fixed the problem, and also: * split bub.d up into several smaller files, * added a Makefile to bootstrap the building of bub, and * added a bub.cfg and Bubfile as a trivial example of how to use bub.Please add an "install:" target to the Makefile. Thank you.
Jul 02 2013
On Tue, 02 Jul 2013 14:07:16 +0200, eles wrote:On Monday, 1 July 2013 at 03:26:22 UTC, Graham St Jack wrote:A better makefile is in progress, and it will have an install target.On Thu, 27 Jun 2013 22:49:41 +0000, Graham St Jack wrote: Fixed the problem, and also: * split bub.d up into several smaller files, * added a Makefile to bootstrap the building of bub, and * added a bub.cfg and Bubfile as a trivial example of how to use bub.Please add an "install:" target to the Makefile. Thank you.
Jul 02 2013
Hm, bub.. Sounds like it should work with 'dub' nicely ;) Looks promising and I'd really love to see some build tool other then rdmd getting to the point it can be called standard. Makefile's sometimes are just too inconvenient.
Jun 27 2013
On Thu, 27 Jun 2013 12:24:27 +0200, Dicebot wrote:Hm, bub.. Sounds like it should work with 'dub' nicely ;)Maybe so...Looks promising and I'd really love to see some build tool other then rdmd getting to the point it can be called standard. Makefile's sometimes are just too inconvenient.This isn't a build tool for everyone, but it really does make a big difference on big projects.
Jun 27 2013
On Thursday, 27 June 2013 at 23:03:40 UTC, Graham St Jack wrote:This isn't a build tool for everyone, but it really does make a big difference on big projects.Well I'm noticing some interesting concepts, such as being able to associate an include or import file with the library to link in, so that it gets done automatically simply by using the include/import file, great idea assuming I understood that correctly. One thing you may want to consider, or maybe this is already possible, is add the ability to optionally specify something like *.d so that all .d files get included from a folder, rather than have to always individually specify them manually. Also more concise documentation with clearer examples would be invaluable. I'm currently unsure if I need to restructure my existing project folders to fit bub or if bub can be configured to fit my existing projects. --rt
Jun 27 2013
On Fri, 28 Jun 2013 08:05:08 +0200, Rob T wrote:On Thursday, 27 June 2013 at 23:03:40 UTC, Graham St Jack wrote:That is correct. A side effect of assigning headers to libraries is that bub can then enforce the visibility rules and give understandable error messages almost instantaneously when you transgress them. Another consequence is that the compiler only needs to be given a few -I directives because you aren't using them to control visibility.This isn't a build tool for everyone, but it really does make a big difference on big projects.Well I'm noticing some interesting concepts, such as being able to associate an include or import file with the library to link in, so that it gets done automatically simply by using the include/import file, great idea assuming I understood that correctly.One thing you may want to consider, or maybe this is already possible, is add the ability to optionally specify something like *.d so that all .d files get included from a folder, rather than have to always individually specify them manually.A useful idea. I will add it to the list.Also more concise documentation with clearer examples would be invaluable. I'm currently unsure if I need to restructure my existing project folders to fit bub or if bub can be configured to fit my existing projects.Documentation is always tough to pitch at the right level. I will see what I can do. Whether you need to restructure your project depends on whether you have already thought about dependencies and arranged things accordingly. My suggestion is to extend the example a bit and get a feel for how it works, then think about what that means for your project.
Jun 27 2013
On 6/26/13 5:10 PM, Graham St Jack wrote:Bottom-up-build (bub) is a build system written in D which supports building of large C/C++/D projects.http://www.reddit.com/r/programming/comments/1h6p2w/bottomupbuild_a_build_system_for_ccd/ Andrei
Jun 27 2013
How does this build tool handle projects with multiple executables ? For example the util-linux package contains dozens of utilities or a project might have a CLI and a GUI version. Or there might be slight alterations like setting a version or debug flag: -debug=threading -version=demo -- Marco
Jun 27 2013
On Fri, 28 Jun 2013 08:28:03 +0200, Marco Leise wrote:How does this build tool handle projects with multiple executables ? For example the util-linux package contains dozens of utilities or a project might have a CLI and a GUI version. Or there might be slight alterations like setting a version or debug flag: -debug=threading -version=demoA central theme in bottom-up-build is to produce any number of build artifacts: libraries, executables, tests, etc. Different versions are catered for with: bub-config --mode=<whatever> <build-dir> The mode is described in your configuration file, and results in a build directory set up specifically for that mode. If you want multiple modes, use multiple build directories. Take a look in example/bub.cfg.
Jun 28 2013
Am Fri, 28 Jun 2013 07:03:27 +0000 (UTC) schrieb Graham St Jack <graham.stjack internode.on.net>:On Fri, 28 Jun 2013 08:28:03 +0200, Marco Leise wrote:Sounds good! -- MarcoHow does this build tool handle projects with multiple executables ? For example the util-linux package contains dozens of utilities or a project might have a CLI and a GUI version. Or there might be slight alterations like setting a version or debug flag: -debug=threading -version=demoA central theme in bottom-up-build is to produce any number of build artifacts: libraries, executables, tests, etc. Different versions are catered for with: bub-config --mode=<whatever> <build-dir> The mode is described in your configuration file, and results in a build directory set up specifically for that mode. If you want multiple modes, use multiple build directories. Take a look in example/bub.cfg.
Jun 28 2013
On 27/06/13 09:40, Graham St Jack wrote:Bottom-up-build (bub) is a build system written in D which supports building of large C/C++/D projects.I've worked with Graham in the past and can attest to bub's coolness. My pet project 'terminol' is about a month away from an initial release, but given that is uses the bub build system, and that Graham has made his announcement, it could be interest now to anyone wishing to see a small project that uses bub: https://github.com/bagnose/terminol Briefly: terminol is a lightweight terminal emulator for Linux, essentially aimed at being a rxvt-unicode refresh. It should checkout, build fine with bub, but so far I'm the only one who ever has. I'm sorry, it's not written in D (it's written in C++11) but I am a huge fan of D. Having worked with GNU Make, Jam and autotools (and looked closely at several others) I find bub very pleasant to use indeed. I won't repeat the benefits that Graham already espoused. I think the focus with bub thus far has been to develop its core approaches: unifying C/C++/D, bottom-up-build, library inference, circularity detection, etc. As such, it is a little unfurnished at the edges. For example, bub does not have a defined scheme for determining things like: what libraries does the user have available and what are their peculiarities (versions, location, etc). At the moment the user may need to do a bit of bub.cfg customisation of their own to make things work. Those things are the next hurdle and will be interesting to discuss. However I think bub already demonstrates the promise of greatness and is worth taking a good look at. David Bryant
Jun 28 2013
On Thursday, 27 June 2013 at 00:10:37 UTC, Graham St Jack wrote:Having side-by-side comparisons of D against bash scripts and C++ modules had the effect of turning almost all the other team members into D advocates.Any chance we could know what team this is? (Sorry if this is common knowledge)
Jun 28 2013
On Sat, 29 Jun 2013 00:59:15 +0200, John Colvin wrote:On Thursday, 27 June 2013 at 00:10:37 UTC, Graham St Jack wrote:It is the development team at my previous workplace. I haven't asked them for permission, so I would rather not say who they are. David Bryant (the previous poster) was a member of that team though, and he will be happy to provide details.Having side-by-side comparisons of D against bash scripts and C++ modules had the effect of turning almost all the other team members into D advocates.Any chance we could know what team this is? (Sorry if this is common knowledge)
Jun 29 2013
On Saturday, 29 June 2013 at 07:54:30 UTC, Graham St Jack wrote:On Sat, 29 Jun 2013 00:59:15 +0200, John Colvin wrote:I am still part of that team. We still use the early predecessor of bub to build several projects from a large C++ codebase, but I am looking forward to switching to bub at the end of the current project. For example, one project uses ~2000 cc/h files containing ~390,000 lines (before any code generation). After a successfull build, running the build again takes about 2s to figure out that everything is up to date. Knowing both build tools, I expect that time to remain about the same after the switch to bub. Yes, I have become a D advocate. Our production code is still mostly C++, but a large part of our system-level testing is now written in D (~15,000 lines). Bash scripts are becoming rarer too. :)On Thursday, 27 June 2013 at 00:10:37 UTC, Graham St Jack wrote:It is the development team at my previous workplace. I haven't asked them for permission, so I would rather not say who they are. David Bryant (the previous poster) was a member of that team though, and he will be happy to provide details.Having side-by-side comparisons of D against bash scripts and C++ modules had the effect of turning almost all the other team members into D advocates.Any chance we could know what team this is? (Sorry if this is common knowledge)
Jun 29 2013
Hooo, a self-contained build tool? That's cool. 1. Are arbitrary make-style commands supported? For example, on windows one may want to compile resources. Resources consist of a declaration .rc file, icons and manifest files, which are compiled into .res file, though only .rc file is passed to the resource compiler, but other files are still dependencies of course. 2. How to do heterogeneous linking? Again on windows an executable can be linked from code .obj's, resource .res and module definition .def: gcc main.o r.res mytool.def -o mytool.exe 3. Probably not actual for big projects. Can build workspase be created automatically by bub instead of explicitly by bub-config... for some default mode? 4. If a build server does builds from scratch, shouldn't it be better for performance to compile several source files in one command? Also applies to user builds: when one wants to install a project from source, he usually does it only once and from scratch. Have you any idea, how this affects compilation speed of c, c++ and d? Though dmd frontend can go out of memory.
Jul 01 2013
On Mon, 01 Jul 2013 18:32:54 +0200, Kagamin wrote:Hooo, a self-contained build tool? That's cool.I like to think so.1. Are arbitrary make-style commands supported? For example, on windows one may want to compile resources. Resources consist of a declaration .rc file, icons and manifest files, which are compiled into .res file, though only .rc file is passed to the resource compiler, but other files are still dependencies of course.Not completely arbitrary. Yous define (currently one-line) commands in bub.cfg, with access to variables that are expanded at the last moment before the command is issued. Some of the variables are pre-defined and populated by bub, and others you define yourself.2. How to do heterogeneous linking? Again on windows an executable can be linked from code .obj's, resource .res and module definition .def: gcc main.o r.res mytool.def -o mytool.exeThere isn't any fundamental reason why things like resource files can't be linked in, but I would probably have to make a few improvements. The use case that bub was forged in was linux-hosted C/C++ with lots of code generated from IDL files and the like.3. Probably not actual for big projects. Can build workspase be created automatically by bub instead of explicitly by bub-config... for some default mode?You set up a build directory with bub-config. I can't currently see any reason why it would be useful for bub to invoke bub-config itself. Re big projects - I guess it depends on what you mean by big. See the post from Clive Hobson, which describes a project that bub works well on. That project seems pretty big to me.4. If a build server does builds from scratch, shouldn't it be better for performance to compile several source files in one command? Also applies to user builds: when one wants to install a project from source, he usually does it only once and from scratch. Have you any idea, how this affects compilation speed of c, c++ and d? Though dmd frontend can go out of memory.I assume you mean my preference for explicitly building object files. I like this because: * It opens up more opportunities for using multiple cores (-j9) * Dependency/reference tracking is simplified. * Test utilities are easier if the code under test is available in a lib.
Jul 02 2013
On Monday, 1 July 2013 at 16:32:55 UTC, Kagamin wrote:4. If a build server does builds from scratch, shouldn't it be better for performance to compile several source files in one command? Also applies to user builds: when one wants to install a project from source, he usually does it only once and from scratch. Have you any idea, how this affects compilation speed of c, c++ and d? Though dmd frontend can go out of memory.I think this should be properly solved by making compiler a daemon/server, not by a build tool.
Jul 02 2013