www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Common makefile (gasp) stuff

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Hey folks,


Over the time a number of stuff has become quite duplicated across our 
makefiles for dmd, druntime, and phobos.

These include fetching OS and model but (newer) general-purpose macros 
for e.g. rebasing repos, see 
https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L37.

I think it's time to reduce clutter and duplication by migrating such 
common stuff into one common.mak file. Question is, where should that 
file sit? One candidate is tools/ but it's not really a tool. Another 
candidate is dmd/src/ because that would be the first thing anyone 
depends on.

Ideas?


Andrei
Jan 13 2015
next sibling parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 13 Jan 2015 15:20:12 -0800
Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
wrote:

 Hey folks,
=20
=20
 Over the time a number of stuff has become quite duplicated across our=20
 makefiles for dmd, druntime, and phobos.
=20
 These include fetching OS and model but (newer) general-purpose macros=20
 for e.g. rebasing repos, see=20
 https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak=
#L37.
=20
 I think it's time to reduce clutter and duplication by migrating such=20
 common stuff into one common.mak file. Question is, where should that=20
 file sit? One candidate is tools/ but it's not really a tool. Another=20
 candidate is dmd/src/ because that would be the first thing anyone=20
 depends on.
=20
 Ideas?
i thing that dmd is the right place. there is little sense in building brand new phobos without brand new dmd with all fixes and enhancements anyway, so i believe that most people who wants to rebuild phobos and druntime are already has dmd repo cloned. besides, this is a basic thing, which is of almost no use for ldc and gdc anyway. i vote for dmd.
Jan 13 2015
prev sibling next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Tue, Jan 13, 2015 at 03:20:12PM -0800, Andrei Alexandrescu via Digitalmars-d
wrote:
 Hey folks,
 
 
 Over the time a number of stuff has become quite duplicated across our
 makefiles for dmd, druntime, and phobos.
 
 These include fetching OS and model but (newer) general-purpose macros for
 e.g. rebasing repos, see
https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L37.
 
 I think it's time to reduce clutter and duplication by migrating such
 common stuff into one common.mak file. Question is, where should that
 file sit? One candidate is tools/ but it's not really a tool. Another
 candidate is dmd/src/ because that would be the first thing anyone
 depends on.
 
 Ideas?
[...] Actually, my first reaction is to go the opposite direction. Currently, we have way too many complicated interdependencies between different repos. For example, the 'html' target of phobos/posix.mak depends on files in the dlang.org repos, and the website build scripts of dlang.org in turn depends on files generated by the Phobos ddoc targets. This may be fine for the core devs and hard-core bleeding-edge people like myself who don't mind running off git HEAD and dealing with breakages, but it's bad for anybody else: 1) These interdependencies are currently expressed by hard-coded filesystem paths, which presumes a specific directory layout for checking out dmd, dlang.org, phobos, druntime. This means the build will break if somebody doesn't know the correct directory structure to use when checking out these repos. 2) This makes the build fragile, because any changes to makefiles that might affect these dependencies have to be merged in sync between multiple repos, e.g., a PR that changes the layout of dlang.org needs to have a corresponding PR that updates the Phobos makefiles as well. This is kinda what we already have to do with compiler-dependent changes to druntime/phobos anyway, but in the case of documentation, any oversight is not caught by the autotester and may wind up on the public-facing dlang.org website before the problem is noticed. 3) Due to these interdependencies, you always have to checkout all 4 repos (in a specific directory layout), otherwise the build will fail one way or another. Again, not a problem for core devs, but a gratuitous obstacle for the casual user who wants to try things out. (Actually, to this day I'm almost certain I *still* don't have the exact layout assumed by the build scripts, because dlang.org appears to deposit generated html files in dlang.org/web, whereas phobos/posix.mak's 'html' target generates them in ../web/, which in my setup is one directory level off. I hacked around the problem with a symlink from web to dlang.org/web, but I'm pretty sure many users who wanted to experiment with Phobos docs gave up because they didn't have the patience to figure out how to get the thing to work properly.) Because of this, my inclination right now is to *reduce* interdependencies between these repos, rather than introduce yet more. Either that, or we should just forget about trying to manually keep 3-4 disparate repos in sync, and just use a single repo for dmd, druntime, phobos, dlang.org, etc.. Except that some people will then complain that they just want the bare compiler and nothing else, so why should they have to clone excess baggage like druntime/phobos. *shrug* Some days you just can't win. (Of course, the *ideal* solution is that the makefiles will tell you if something is missing and where to get it and how to set it up correctly. But I'm skeptical whether this will ever materialize. I'm not holding my breath.) T -- Why waste time reinventing the wheel, when you could be reinventing the engine? -- Damian Conway
Jan 13 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/13/15 3:55 PM, H. S. Teoh via Digitalmars-d wrote:
 On Tue, Jan 13, 2015 at 03:20:12PM -0800, Andrei Alexandrescu via
Digitalmars-d wrote:
 Hey folks,


 Over the time a number of stuff has become quite duplicated across our
 makefiles for dmd, druntime, and phobos.

 These include fetching OS and model but (newer) general-purpose macros for
 e.g. rebasing repos, see
https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L37.

 I think it's time to reduce clutter and duplication by migrating such
 common stuff into one common.mak file. Question is, where should that
 file sit? One candidate is tools/ but it's not really a tool. Another
 candidate is dmd/src/ because that would be the first thing anyone
 depends on.

 Ideas?
[...] Actually, my first reaction is to go the opposite direction. Currently, we have way too many complicated interdependencies between different repos. For example, the 'html' target of phobos/posix.mak depends on files in the dlang.org repos, and the website build scripts of dlang.org in turn depends on files generated by the Phobos ddoc targets.
Yah but when you look closer it's not that bad; just an attribution tradeoff. Building the phobos docs belongs with the phobos makefile because that's where the module informations resides; however, part of the build requires macro definitions that are at best consolidated in the website directory. Building phobos's docs without dlang.org around is not sensible, just the same as building dlang.org without phobos. So I'm not too worried about that. If someone builds phobos without caring about the docs, that's fine as long as they don't try to build html. One simplification would be to not set STDDOC in https://github.com/D-Programming-Language/phobos/blob/master/posix.mak#L95, and refuse to build html if STDDOC is not set from the outside. But again I see this as just minor progress.
 This may
 be fine for the core devs and hard-core bleeding-edge people like myself
 who don't mind running off git HEAD and dealing with breakages, but it's
 bad for anybody else:

 1) These interdependencies are currently expressed by hard-coded
 filesystem paths, which presumes a specific directory layout for
 checking out dmd, dlang.org, phobos, druntime. This means the build will
 break if somebody doesn't know the correct directory structure to use
 when checking out these repos.
Yah, that should be in a wiki. I keep on thinking about producing a 10-minutes video "Your First Contribution to D" in which I explain step by step how to get a pull request in starting from a scratch. We also have tools/update.sh which should probably be more prominently advertised. But in the grand scheme of things I don't see a foolproof solution to that - you need dmd for building druntime, druntime for building phobos, etc. If you got dmd, druntime, and phobos in the wrong places then You're Gonna Have A Bad Time.
 2) This makes the build fragile, because any changes to makefiles that
 might affect these dependencies have to be merged in sync between
 multiple repos, e.g., a PR that changes the layout of dlang.org needs to
 have a corresponding PR that updates the Phobos makefiles as well. This
 is kinda what we already have to do with compiler-dependent changes to
 druntime/phobos anyway, but in the case of documentation, any oversight
 is not caught by the autotester and may wind up on the public-facing
 dlang.org website before the problem is noticed.
I don't understand how this is a problem, how my suggestion makes it any worse, what a solution to it all would be.
 3) Due to these interdependencies, you always have to checkout all 4
 repos (in a specific directory layout), otherwise the build will fail
 one way or another.
This is expected, no? What is an alternative? How would one e.g. build druntime without dmd?
 Again, not a problem for core devs, but a gratuitous
 obstacle for the casual user who wants to try things out. (Actually, to
 this day I'm almost certain I *still* don't have the exact layout
 assumed by the build scripts, because dlang.org appears to deposit
 generated html files in dlang.org/web, whereas phobos/posix.mak's 'html'
 target generates them in ../web/, which in my setup is one directory
 level off. I hacked around the problem with a symlink from web to
 dlang.org/web, but I'm pretty sure many users who wanted to experiment
 with Phobos docs gave up because they didn't have the patience to figure
 out how to get the thing to work properly.)
That's an inconsistency we should fix. Right now phobos is best build starting from the dlang.org build. We should probably disable phobos' own docs build.
 Because of this, my inclination right now is to *reduce*
 interdependencies between these repos, rather than introduce yet more.

 Either that, or we should just forget about trying to manually keep 3-4
 disparate repos in sync, and just use a single repo for dmd, druntime,
 phobos, dlang.org, etc.. Except that some people will then complain that
 they just want the bare compiler and nothing else, so why should they
 have to clone excess baggage like druntime/phobos. *shrug* Some days you
 just can't win.
Yah, that's why I think a set of simple tactical tools a la tools/update.sh are the proportional response here.
 (Of course, the *ideal* solution is that the makefiles will tell you if
 something is missing and where to get it and how to set it up correctly.
 But I'm skeptical whether this will ever materialize.  I'm not holding
 my breath.)
dlang.org does something of that kind already - "make rebase". Andrei
Jan 13 2015
parent reply Paul O'Neil <redballoon36 gmail.com> writes:
On 01/13/2015 09:35 PM, Andrei Alexandrescu wrote:
<snip>
 1) These interdependencies are currently expressed by hard-coded
 filesystem paths, which presumes a specific directory layout for
 checking out dmd, dlang.org, phobos, druntime. This means the build will
 break if somebody doesn't know the correct directory structure to use
 when checking out these repos.
Yah, that should be in a wiki. I keep on thinking about producing a 10-minutes video "Your First Contribution to D" in which I explain step by step how to get a pull request in starting from a scratch. We also have tools/update.sh which should probably be more prominently advertised.
<snip> This shouldn't go (only) in the wiki. It should be in the README of every repository. And if not in the README, then the CONTRIBUTING file. A minimal list of "clone these things, run make in these directories" is fine here with a reference to the wiki for more details. I put off some documentation updates a few months ago because it wasn't worth the investment yet to figure out how to build the website. The wiki pages appeared to be about building the compiler, which I didn't realize I should be interested in. This barrier is too high. -- Paul O'Neil Github / IRC: todayman
Jan 13 2015
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/13/15 8:29 PM, Paul O'Neil wrote:
 On 01/13/2015 09:35 PM, Andrei Alexandrescu wrote:
 <snip>
 1) These interdependencies are currently expressed by hard-coded
 filesystem paths, which presumes a specific directory layout for
 checking out dmd, dlang.org, phobos, druntime. This means the build will
 break if somebody doesn't know the correct directory structure to use
 when checking out these repos.
Yah, that should be in a wiki. I keep on thinking about producing a 10-minutes video "Your First Contribution to D" in which I explain step by step how to get a pull request in starting from a scratch. We also have tools/update.sh which should probably be more prominently advertised.
<snip> This shouldn't go (only) in the wiki. It should be in the README of every repository. And if not in the README, then the CONTRIBUTING file. A minimal list of "clone these things, run make in these directories" is fine here with a reference to the wiki for more details. I put off some documentation updates a few months ago because it wasn't worth the investment yet to figure out how to build the website. The wiki pages appeared to be about building the compiler, which I didn't realize I should be interested in. This barrier is too high.
Great ideas! I created https://issues.dlang.org/show_bug.cgi?id=13980 to track them. Thanks, Andrei
Jan 13 2015
prev sibling parent reply "Mike" <none none.com> writes:
On Wednesday, 14 January 2015 at 04:29:57 UTC, Paul O'Neil wrote:
 I put off some documentation updates a few months ago because 
 it wasn't
 worth the investment yet to figure out how to build the 
 website.  The
 wiki pages appeared to be about building the compiler, which I 
 didn't
 realize I should be interested in.  This barrier is too high.
I know the feeling. I finally figured out how to get a single ddoc file to html so I could help update the documentation. I added a few notes to the dlang.org CONTRIBUTING.md [1] file a while back to help other users with that. It's probably not very noticeable there, but I don't understand the build process well enough to do a proper job. [1] https://github.com/D-Programming-Language/dlang.org/blob/master/CONTRIBUTING.md Why does documentation need to be "built" anyway. I understand the autogenerated stuff from the source code, but not the website. Would it be better to wikify it, somehow, in GitHub pages? Mike
Jan 13 2015
parent "Mike" <none none.com> writes:
On Wednesday, 14 January 2015 at 07:55:31 UTC, Mike wrote:

 Would it be better to wikify it, somehow, in GitHub
 pages?
I think wiki is the wrong term here. I mean, is there a way to leverage Markdown (or some other markup language) with GitHub's source code management features to make it easier for contributors to fork, edit in their browser, verify in their repository, and create pull requests without having to build with a compiler? Mike
Jan 14 2015
prev sibling next sibling parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Andrei Alexandrescu"  wrote in message 
news:m9497d$1pv2$1 digitalmars.com...

 Over the time a number of stuff has become quite duplicated across our 
 makefiles for dmd, druntime, and phobos.

 These include fetching OS and model but (newer) general-purpose macros for 
 e.g. rebasing repos, see 
 https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L37.

 I think it's time to reduce clutter and duplication by migrating such 
 common stuff into one common.mak file. Question is, where should that file 
 sit? One candidate is tools/ but it's not really a tool. Another candidate 
 is dmd/src/ because that would be the first thing anyone depends on.

 Ideas?
If it's optional extra stuff like git shortcuts, then removing it from the makefiles and putting it in tools makes sense. If it's mandatory stuff like OS detection, then it needs to stay where it is. What's the actual problem you're trying to solve with this? IMO duplication is less of a problem than overcomplicating the makefiles.
Jan 13 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/13/15 7:44 PM, Daniel Murphy wrote:
 "Andrei Alexandrescu"  wrote in message
 news:m9497d$1pv2$1 digitalmars.com...

 Over the time a number of stuff has become quite duplicated across our
 makefiles for dmd, druntime, and phobos.

 These include fetching OS and model but (newer) general-purpose macros
 for e.g. rebasing repos, see
 https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L37.


 I think it's time to reduce clutter and duplication by migrating such
 common stuff into one common.mak file. Question is, where should that
 file sit? One candidate is tools/ but it's not really a tool. Another
 candidate is dmd/src/ because that would be the first thing anyone
 depends on.

 Ideas?
If it's optional extra stuff like git shortcuts, then removing it from the makefiles and putting it in tools makes sense. If it's mandatory stuff like OS detection, then it needs to stay where it is.
Hmmm... I wonder why the distinction.
 What's the actual problem you're trying to solve with this?  IMO
 duplication is less of a problem than overcomplicating the makefiles.
The problem is basic code duplication with its known liabilities. I'm looking at stuff like this: https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L58 https://github.com/D-Programming-Language/phobos/blob/master/posix.mak#L31 https://github.com/D-Programming-Language/druntime/blob/master/posix.mak#L8 Three's a charm. That code is relatively verbose (required newlines). I trust it doesn't need updates often, but it occupies real estate in a readily accessible position (beginning of file). The real issue here is that acquiescing to such duplication discourages attempts at uniformization before they occur. I'm thinking of simpler, clearer naming conventions and uniform ways of doing the same things (such as unittesting or documentation). Getting back to REBASE (https://github.com/D-Programming-Language/dlang.org/blob/ma ter/posix.mak#L37), it greatly simplifies my building the entire website in a way I trust is nice: make rebase -j && make clean && make rsync -j (Sadly make clean is necessary because of dub, which makes the process take a lot longer than it should; with luck, somebody will improve on that soon.) Now "make rebase" is really useful for druntime and phobos as well. But between duplicating more code or just doing it by hand, one may as well give up and choose the latter. I'd also like to define "make rebase-dirty" based on this: REBASE_DIRTY = MYBRANCH=`git rev-parse --abbrev-ref HEAD` &&\ git stash &&\ git co master &&\ git pull --ff-only upstream master &&\ git co $$MYBRANCH &&\ git rebase master &&\ git stash pop which is useful because it lets me rebase on top of work in progress. Again the prospect of duplicating this other macro across 3-4 projects is fairly unattractive. Andrei
Jan 13 2015
parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Andrei Alexandrescu"  wrote in message 
news:m94vfk$2gdh$1 digitalmars.com...

 Hmmm... I wonder why the distinction.
OS detection is required to build successfully, a rebase shortcut is not.
 The problem is basic code duplication with its known liabilities. I'm 
 looking at stuff like this:

 Three's a charm. That code is relatively verbose (required newlines). I 
 trust it doesn't need updates often, but it occupies real estate in a 
 readily accessible position (beginning of file).

 The real issue here is that acquiescing to such duplication discourages 
 attempts at uniformization before they occur. I'm thinking of simpler, 
 clearer naming conventions and uniform ways of doing the same things (such 
 as unittesting or documentation).
The problem here is really that make sucks, right? And makefiles suck. Maybe we should start seriously looking at replacing them with D scripts.
 Getting back to REBASE 
 (https://github.com/D-Programming-Language/dlang.org/blob/ma
ter/posix.mak#L37), 
 it greatly simplifies my building the entire website in a way I trust is 
 nice:

    make rebase -j && make clean && make rsync -j

 (Sadly make clean is necessary because of dub, which makes the process 
 take a lot longer than it should; with luck, somebody will improve on that 
 soon.)

 Now "make rebase" is really useful for druntime and phobos as well. But 
 between duplicating more code or just doing it by hand, one may as well 
 give up and choose the latter.
Honestly I don't think automating rebase is a good idea, because it discourages actually learning how to do it. eg if all you're trying to do is get the current branch rebased on top of upstream's master, you can do this: git pull upstream master --rebase
 I'd also like to define "make rebase-dirty" based on this:

 REBASE_DIRTY = MYBRANCH=`git rev-parse --abbrev-ref HEAD` &&\
   git stash &&\
   git co master &&\
   git pull --ff-only upstream master &&\
   git co $$MYBRANCH &&\
   git rebase master &&\
   git stash pop

 which is useful because it lets me rebase on top of work in progress. 
 Again the prospect of duplicating this other macro across 3-4 projects is 
 fairly unattractive.
I've managed to get by fine with using shell scripts for stuff like this.
Jan 14 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/14/15 12:29 AM, Daniel Murphy wrote:
 The problem here is really that make sucks, right?  And makefiles suck.
 Maybe we should start seriously looking at replacing them with D scripts.
I agree that make sucks etc. but duplication sucks more (and would suck regardless of what tool we use). Here we do have a simple method (include files) that is suitable for addressing that. I'd assume we'd use something similar with whichever tool we'd use.
 Honestly I don't think automating rebase is a good idea, because it
 discourages actually learning how to do it.

 eg if all you're trying to do is get the current branch rebased on top
 of upstream's master, you can do this:

 git pull upstream master --rebase
Great. As an aside I just ran that and got: error: unknown option `rebase' usage: ... Whether it's a command or a couple, that doesn't change the fact that I need to run the same command four times, and in parallel if at all possible. It's a perfect candidate for automation, and I haven't seen an argument the makefile isn't a good place for that.
 I'd also like to define "make rebase-dirty" based on this:

 REBASE_DIRTY = MYBRANCH=`git rev-parse --abbrev-ref HEAD` &&\
   git stash &&\
   git co master &&\
   git pull --ff-only upstream master &&\
   git co $$MYBRANCH &&\
   git rebase master &&\
   git stash pop

 which is useful because it lets me rebase on top of work in progress.
 Again the prospect of duplicating this other macro across 3-4 projects
 is fairly unattractive.
I've managed to get by fine with using shell scripts for stuff like this.
Do you need to upload the dlang.org website quickly and without error? Overall I have difficulty understanding counterarguments. Challenge: "We should eliminate some unpleasant duplication." Response: "Make sucks. Make includes are unreliable for mythical unexplained reasons. You don't know git. You should do scripts instead." The response just doesn't follow. Care to clarify what you think we should be doing here? Andrei
Jan 14 2015
parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Andrei Alexandrescu"  wrote in message 
news:m95ds3$2vm4$1 digitalmars.com...

 I agree that make sucks etc. but duplication sucks more (and would suck 
 regardless of what tool we use). Here we do have a simple method (include 
 files) that is suitable for addressing that. I'd assume we'd use something 
 similar with whichever tool we'd use.
I don't mind using include files, I mind adding more dependencies between projects.
 Great. As an aside I just ran that and got:

 error: unknown option `rebase'
 usage: ...
Maybe it's 'git pull --rebase upstream master', I didn't try it and I don't type it manually very often because I have a script to do this.
 Whether it's a command or a couple, that doesn't change the fact that I 
 need to run the same command four times, and in parallel if at all 
 possible. It's a perfect candidate for automation, and I haven't seen an 
 argument the makefile isn't a good place for that.
I guess we have different usage patterns, which isn't really a surprise. Anyway putting rebase in the makefile is a secondary issue, that's probably not worth discussing now.
 I've managed to get by fine with using shell scripts for stuff like 
 this.
Do you need to upload the dlang.org website quickly and without error?
Do enough people need to that it warrants being in the makefile instead of a local shell script? Does everyone do it the same way?
 Overall I have difficulty understanding counterarguments.

 Challenge: "We should eliminate some unpleasant duplication."

 Response: "Make sucks. Make includes are unreliable for mythical 
 unexplained reasons. You don't know git. You should do scripts instead."

 The response just doesn't follow. Care to clarify what you think we should 
 be doing here?
Apologies for not being clear. Challenge: "We should eliminate some unpleasant duplication." Response: "We should avoid adding more dependencies between projects at all costs."
Jan 14 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/14/15 2:09 AM, Daniel Murphy wrote:
 "Andrei Alexandrescu"  wrote in message
 news:m95ds3$2vm4$1 digitalmars.com...

 I agree that make sucks etc. but duplication sucks more (and would
 suck regardless of what tool we use). Here we do have a simple method
 (include files) that is suitable for addressing that. I'd assume we'd
 use something similar with whichever tool we'd use.
I don't mind using include files, I mind adding more dependencies between projects.
Why? Druntime depends on dmd. Phobos depends on dmd and druntime. Dlang.org and tools depend on dmd, druntime, and phobos. It's a fact of life.
 Great. As an aside I just ran that and got:

 error: unknown option `rebase'
 usage: ...
Maybe it's 'git pull --rebase upstream master', I didn't try it and I don't type it manually very often because I have a script to do this.
 Whether it's a command or a couple, that doesn't change the fact that
 I need to run the same command four times, and in parallel if at all
 possible. It's a perfect candidate for automation, and I haven't seen
 an argument the makefile isn't a good place for that.
I guess we have different usage patterns, which isn't really a surprise. Anyway putting rebase in the makefile is a secondary issue, that's probably not worth discussing now.
 I've managed to get by fine with using shell scripts for stuff like
 this.
Do you need to upload the dlang.org website quickly and without error?
Do enough people need to that it warrants being in the makefile instead of a local shell script? Does everyone do it the same way?
Yes. Currently I'm the site build master by default. In the near future there'll be several. I'd rather have anyone run a simple command than telling them to write their scripts. So if next month I transfer this responsibility to Martin, I can tell: "to update dlang.org, run 'make rebase -j && make rsync -j'" as opposed to "yeah, I have some scripts somewhere, do your own or let me email them to you". I don't really understand your position here. It's like you reject robust automation over tribal knowledge and folklore.
 Overall I have difficulty understanding counterarguments.

 Challenge: "We should eliminate some unpleasant duplication."

 Response: "Make sucks. Make includes are unreliable for mythical
 unexplained reasons. You don't know git. You should do scripts instead."

 The response just doesn't follow. Care to clarify what you think we
 should be doing here?
Apologies for not being clear. Challenge: "We should eliminate some unpleasant duplication." Response: "We should avoid adding more dependencies between projects at all costs."
Why? As an aside, "at all costs" is suspicious in many contexts. Andrei
Jan 14 2015
next sibling parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Andrei Alexandrescu"  wrote in message news:m95i3q$1t6$1 digitalmars.com...

 I don't mind using include files, I mind adding more dependencies
 between projects.
Why? Druntime depends on dmd. Phobos depends on dmd and druntime. Dlang.org and tools depend on dmd, druntime, and phobos. It's a fact of life.
Druntime, phobos and dlang.org don't (currently) depend on dmd's source.
Jan 14 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/14/15 3:29 AM, Daniel Murphy wrote:
 "Andrei Alexandrescu"  wrote in message
 news:m95i3q$1t6$1 digitalmars.com...

 I don't mind using include files, I mind adding more dependencies
 between projects.
Why? Druntime depends on dmd. Phobos depends on dmd and druntime. Dlang.org and tools depend on dmd, druntime, and phobos. It's a fact of life.
Druntime, phobos and dlang.org don't (currently) depend on dmd's source.
I see your point, though I'll note that DMD everywhere defaults to ../dmd/src/dmd (and that's a huge improvement from the time is was defined as just dmd). But granted setting it in the command line would make those independent of the dmd location. So the question is - is there a place that's common to all? Andrei
Jan 14 2015
prev sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Wed, Jan 14, 2015 at 02:58:02AM -0800, Andrei Alexandrescu via Digitalmars-d
wrote:
 On 1/14/15 2:09 AM, Daniel Murphy wrote:
[...]
Do enough people need to that it warrants being in the makefile
instead of a local shell script?  Does everyone do it the same way?
Yes. Currently I'm the site build master by default. In the near future there'll be several. I'd rather have anyone run a simple command than telling them to write their scripts. So if next month I transfer this responsibility to Martin, I can tell: "to update dlang.org, run 'make rebase -j && make rsync -j'" as opposed to "yeah, I have some scripts somewhere, do your own or let me email them to you". I don't really understand your position here. It's like you reject robust automation over tribal knowledge and folklore.
What stops us from adding said scripts to the dlang.org repo? Or tools, for that matter, which is probably more appropriate. Not everything needs to be shoehorned into the makefiles.
Overall I have difficulty understanding counterarguments.

Challenge: "We should eliminate some unpleasant duplication."

Response: "Make sucks. Make includes are unreliable for mythical
unexplained reasons. You don't know git. You should do scripts
instead."

The response just doesn't follow. Care to clarify what you think we
should be doing here?
Apologies for not being clear. Challenge: "We should eliminate some unpleasant duplication." Response: "We should avoid adding more dependencies between projects at all costs."
Why? As an aside, "at all costs" is suspicious in many contexts.
Because it imposes a specific external directory structure which the casual user has no idea about. This stuff is documented nowhere currently -- I had to read the makefiles to figure it out for myself; not many people have that much patience. It makes the whole thing more fragile because as soon as one or more repos go out-of-sync (not on github, but in the user's local clones, which can happen very easily) things can break horribly without any warning. I'm astonished this has to be spelled out in, of all places, a D forum. Surely, of all people, we D people must be all too familiar with exactly why things like #include <../../../mylib/mymod/myheader.h> are evil? I mean, why else would we have D modules rather than continue this glorious C heritage? Yet here we're proposing to add the equivalent of such out-of-tree #include's to our makefiles *and* we're straining to find reasons why we shouldn't do it? I'm at a loss of words to describe the cognitive dissonance here. T -- Creativity is not an excuse for sloppiness.
Jan 14 2015
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/14/15 9:36 AM, H. S. Teoh via Digitalmars-d wrote:
 What stops us from adding said scripts to the dlang.org repo? Or tools,
 for that matter, which is probably more appropriate.

 Not everything needs to be shoehorned into the makefiles.
As I explained in my previous message it's a sensible solution. -- Andrei
Jan 14 2015
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/14/15 9:36 AM, H. S. Teoh via Digitalmars-d wrote:
 Because it imposes a specific external directory structure which the
 casual user has no idea about.
I think that's a given already and has been for a long time. -- Andrei
Jan 14 2015
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/14/15 9:36 AM, H. S. Teoh via Digitalmars-d wrote:
 I'm astonished this has to be spelled out in, of all places, a D forum.
 Surely, of all people, we D people must be all too familiar with exactly
 why things like #include <../../../mylib/mymod/myheader.h> are evil?
I think that would be applying one canned lesson to a situation where it doesn't quite apply. -- Andrei
Jan 14 2015
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/14/15 9:36 AM, H. S. Teoh via Digitalmars-d wrote:
 What stops us from adding said scripts to the dlang.org repo? Or tools,
 for that matter, which is probably more appropriate.
Oh, I just figured that would make everything dependent on dlang.org. That would be a new dependency :o). -- Andrei
Jan 14 2015
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/13/2015 3:20 PM, Andrei Alexandrescu wrote:
 Over the time a number of stuff has become quite duplicated across our
makefiles
 for dmd, druntime, and phobos.

 These include fetching OS and model but (newer) general-purpose macros for e.g.
 rebasing repos, see
 https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L37.

 I think it's time to reduce clutter and duplication by migrating such common
 stuff into one common.mak file. Question is, where should that file sit? One
 candidate is tools/ but it's not really a tool. Another candidate is dmd/src/
 because that would be the first thing anyone depends on.

 Ideas?
It's a few lines of duplication, as opposed to adding another point of failure to the build process, i.e. common.mak isn't found or the wrong one is found, etc. I've always tended to avoid using make macro include's for this reason.
Jan 13 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/13/15 8:49 PM, Walter Bright wrote:
 It's a few lines of duplication, as opposed to adding another point of
 failure to the build process, i.e. common.mak isn't found or the wrong
 one is found, etc.
I'd have difficulty agreeing with this. If dmd is found then whatever is there is found. It's a deterministic process, not an arbitrary path lookup dependent on system vagaries. Plus... duplication only grows.
 I've always tended to avoid using make macro include's for this reason.
I think I failed to grasp the reason. Andrei
Jan 13 2015
prev sibling next sibling parent Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 2015-01-13 at 15:20 -0800, Andrei Alexandrescu via Digitalmars-d
wrote:
 Hey folks,
=20
=20
 Over the time a number of stuff has become quite duplicated across our=
=20
 makefiles for dmd, druntime, and phobos.
Are these three separate things with separate builds, or are they three components of the same system? What is the relationship between these three, LDC and GDC?
 These include fetching OS and model but (newer) general-purpose macros=
=20
 for e.g. rebasing repos, see=20
 https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak=
#L37.
=20
 I think it's time to reduce clutter and duplication by migrating such=20
 common stuff into one common.mak file. Question is, where should that=20
 file sit? One candidate is tools/ but it's not really a tool. Another=20
 candidate is dmd/src/ because that would be the first thing anyone=20
 depends on.
=20
 Ideas?
Are all three things in the same Git repository? Can I clone the three independently or do I have to have them all? Can any of them be built without building the other or does a DMD build require a new druntime and phobos build. Basically what I am trying to say here is that I suspect the extraction of commonality is going to create an inappropriate coupling. The solution is to have the common material as a fourth thing that can be brought in to any or all of the three. With SCons and Waf this is achieved by having a tool, easy simple and keeps the coupling low. I haven't used Make in 12 years, so I have no idea what it can do these days, But unless you mandate GNU Make, I suspect it is going to be awful trying to build across Windows, OSX, Debian, Arch, Fedora, FreeBSD, OpenBSD, etc. without hacking (in the worst sense). --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Jan 14 2015
prev sibling next sibling parent Rikki Cattermole <alphaglosined gmail.com> writes:
One thought because of somethings being said, dmd-fe is being converted 
to D. Which means to compile dmd you need a D compiler already.
We could make all the build scripts, D instead.

It would mean one less external to D ecosystem dependency.
But it could easily be a nightmare.
Jan 14 2015
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-01-14 00:20, Andrei Alexandrescu wrote:
 Hey folks,


 Over the time a number of stuff has become quite duplicated across our
 makefiles for dmd, druntime, and phobos.

 These include fetching OS and model but (newer) general-purpose macros
 for e.g. rebasing repos, see
 https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L37.


 I think it's time to reduce clutter and duplication by migrating such
 common stuff into one common.mak file. Question is, where should that
 file sit? One candidate is tools/ but it's not really a tool. Another
 candidate is dmd/src/ because that would be the first thing anyone
 depends on.
Oh God, please no. I really, really hate that the makefiles are are looking in folders outside of the repositories.
 Ideas?
If you really want to do something like this I would recommend creating a new repository, acting like a super repository, including the other repositories as submodules. Then you can create a new makefile that builds everything. Although, I still don't want the makefiles in the existing repositories be dependent on the new repository. -- /Jacob Carlborg
Jan 14 2015
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Wed, Jan 14, 2015 at 04:27:10PM +0100, Jacob Carlborg via Digitalmars-d
wrote:
 On 2015-01-14 00:20, Andrei Alexandrescu wrote:
[...]
I think it's time to reduce clutter and duplication by migrating such
common stuff into one common.mak file. Question is, where should that
file sit? One candidate is tools/ but it's not really a tool. Another
candidate is dmd/src/ because that would be the first thing anyone
depends on.
Oh God, please no. I really, really hate that the makefiles are are looking in folders outside of the repositories.
+1. Makefiles that reference paths outside the repo are really really evil. They only ever work for the specific setup the author has, and breaks for everybody else.
Ideas?
If you really want to do something like this I would recommend creating a new repository, acting like a super repository, including the other repositories as submodules. Then you can create a new makefile that builds everything. Although, I still don't want the makefiles in the existing repositories be dependent on the new repository.
[...] +1, this is a much better approach than introducing yet more extra-repo hard-coded paths into the makefiles. There are too many of those already. T -- Truth, Sir, is a cow which will give [skeptics] no more milk, and so they are gone to milk the bull. -- Sam. Johnson
Jan 14 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/14/15 9:01 AM, H. S. Teoh via Digitalmars-d wrote:
 On Wed, Jan 14, 2015 at 04:27:10PM +0100, Jacob Carlborg via Digitalmars-d
wrote:
 On 2015-01-14 00:20, Andrei Alexandrescu wrote:
[...]
 I think it's time to reduce clutter and duplication by migrating such
 common stuff into one common.mak file. Question is, where should that
 file sit? One candidate is tools/ but it's not really a tool. Another
 candidate is dmd/src/ because that would be the first thing anyone
 depends on.
Oh God, please no. I really, really hate that the makefiles are are looking in folders outside of the repositories.
+1. Makefiles that reference paths outside the repo are really really evil. They only ever work for the specific setup the author has, and breaks for everybody else.
Currently there must be a place where druntime finds dmd for building purposes. Also phobos needs to find druntime. Also dlang.ord needs to find the source of both phobos and druntime. There's no specific setup of one person as much as a boring: /some/dir/ dmd druntime phobos dlang.org tools I have a hard time making it any simpler.
 Ideas?
If you really want to do something like this I would recommend creating a new repository, acting like a super repository, including the other repositories as submodules. Then you can create a new makefile that builds everything. Although, I still don't want the makefiles in the existing repositories be dependent on the new repository.
[...] +1, this is a much better approach than introducing yet more extra-repo hard-coded paths into the makefiles. There are too many of those already.
Where would that super-repo sit? Andrei
Jan 14 2015
next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Wed, Jan 14, 2015 at 11:50:39AM -0800, Andrei Alexandrescu via Digitalmars-d
wrote:
 On 1/14/15 9:01 AM, H. S. Teoh via Digitalmars-d wrote:
On Wed, Jan 14, 2015 at 04:27:10PM +0100, Jacob Carlborg via Digitalmars-d
wrote:
On 2015-01-14 00:20, Andrei Alexandrescu wrote:
[...]
I think it's time to reduce clutter and duplication by migrating
such common stuff into one common.mak file. Question is, where
should that file sit? One candidate is tools/ but it's not really a
tool. Another candidate is dmd/src/ because that would be the first
thing anyone depends on.
Oh God, please no. I really, really hate that the makefiles are are looking in folders outside of the repositories.
+1. Makefiles that reference paths outside the repo are really really evil. They only ever work for the specific setup the author has, and breaks for everybody else.
Currently there must be a place where druntime finds dmd for building purposes. Also phobos needs to find druntime. Also dlang.ord needs to find the source of both phobos and druntime. There's no specific setup of one person as much as a boring: /some/dir/ dmd druntime phobos dlang.org tools I have a hard time making it any simpler.
I think you're missing the point. The problem is not that a specific setup is required. The problem is that the user has to (1) know exactly what this setup is, and (2) set it up *manually*. The first point is a dire lack of documentation, the second point begs the question "why require the user to do this when we can do this for them?". I.e., via submodules, see below. Not to mention, your proposed setup does not match dlang.org's makefile's idea of what the setup should be. (For example, it generates html files in dlang.org/web whereas the Phobos makefile generates html files in phobos/../web, which appears to suggest that {dmd,druntime,phobos} ought to be nested inside dlang.org instead of being siblings as you have shown.) If even you can't keep things consistent with what's in dlang.org's makefiles, what hope do the rest of us have, let alone a casual user who's only tentatively interested in D?
Ideas?
If you really want to do something like this I would recommend creating a new repository, acting like a super repository, including the other repositories as submodules. Then you can create a new makefile that builds everything. Although, I still don't want the makefiles in the existing repositories be dependent on the new repository.
[...] +1, this is a much better approach than introducing yet more extra-repo hard-coded paths into the makefiles. There are too many of those already.
Where would that super-repo sit?
[...] Huh?? What do you mean, where it would sit? If you set up a "super-repo", it would look something like this: dlang/ # root git repo dmd/ # git submodule referencing the dmd repo druntime/ # git submodule referencing the druntime repo phobos/ # git submodule referencing the phobos repo dlang.org/ # git submodule referencing the dlang.org repo Then checking out this new dlang repo (with the requisite git flags) will create the above structure for you, including clone all the sub-repos. *This* is what will address point (2) that I raised earlier: the user shouldn't need to manually setup some arbitrary directory structure for you -- automate this with git submodules instead! Having such a super-repo will also solve the problem of git bisecting earlier toolchain releases -- because the super-repo references the sub-repos by SHA hashes, which are history-tracked by git, so we could set it up so that each commit of the super-repo corresponds with a valid version set of dmd/druntime/phobos. Then you could checkout a specific commit of the super-repo and update git submodules, and you'll get a snapshot of the entire toolchain as it stood at the time of the commit. If we do this right, we could even have the autotester update the super-repo automatically upon successful completion of the test suite for the entire toolchain. And ideally, stuff common to multiple subrepos should be in this "super-repo", such as the stuff you're trying to factor out right now. Including the master makefile that will build everything WITHOUT needing to reference out-of-tree paths. And ideally, any out-of-tree references needed by the subrepos will be via configurable parameters (e.g. $(DMD), $(DRUNTIME_PATH), etc.), which the master makefile can set to the correct values, so the subrepos don't have to know or hardcode where the exact out-of-tree location ought to be. Such a super-repo will also make release management easier: instead of manually managing release tags in each of dmd/druntime/phobos, we can tag the super-repo, which encapsulates the exact set of dmd/druntime/phobos commits for that release. T -- There are 10 kinds of people in the world: those who can count in binary, and those who can't.
Jan 14 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/14/15 12:18 PM, H. S. Teoh via Digitalmars-d wrote:
 If you set up a
 "super-repo", it would look something like this:

 	dlang/			# root git repo
 		dmd/		# git submodule referencing the dmd repo
 		druntime/	# git submodule referencing the druntime repo
 		phobos/ 	# git submodule referencing the phobos repo
 		dlang.org/	# git submodule referencing the dlang.org repo

 Then checking out this new dlang repo (with the requisite git flags)
 will create the above structure for you, including clone all the
 sub-repos. *This* is what will address point (2) that I raised earlier:
 the user shouldn't need to manually setup some arbitrary directory
 structure for you -- automate this with git submodules instead!
Agreed. Ah, this is getting interesting, thanks. So I've wanting to look into submodules for a while now but have been discouraged by the bad press they seem to be getting: https://codingkilledthecat.wordpress.com/2012/04/28/why-your-company-shouldnt-use-git-submodules/ http://ayende.com/blog/4746/the-problem-with-git-submodules http://blogs.atlassian.com/2013/03/git-submodules-workflows-tips/ http://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree/ http://somethingsinistral.net/blog/git-submodules-are-probably-not-the-answer/ (though I have no idea whether what's relevant for puppet is relevant to us)
 Having such a super-repo will also solve the problem of git bisecting
 earlier toolchain releases -- because the super-repo references the
 sub-repos by SHA hashes, which are history-tracked by git, so we could
 set it up so that each commit of the super-repo corresponds with a valid
 version set of dmd/druntime/phobos. Then you could checkout a specific
 commit of the super-repo and update git submodules, and you'll get a
 snapshot of the entire toolchain as it stood at the time of the commit.
 If we do this right, we could even have the autotester update the
 super-repo automatically upon successful completion of the test suite
 for the entire toolchain.

 And ideally, stuff common to multiple subrepos should be in this
 "super-repo", such as the stuff you're trying to factor out right now.
 Including the master makefile that will build everything WITHOUT needing
 to reference out-of-tree paths. And ideally, any out-of-tree references
 needed by the subrepos will be via configurable parameters (e.g. $(DMD),
 $(DRUNTIME_PATH), etc.), which the master makefile can set to the
 correct values, so the subrepos don't have to know or hardcode where the
 exact out-of-tree location ought to be.

 Such a super-repo will also make release management easier: instead of
 manually managing release tags in each of dmd/druntime/phobos, we can
 tag the super-repo, which encapsulates the exact set of
 dmd/druntime/phobos commits for that release.
I hear you but The Internet says submodules are a kind of a bummer. Andrei
Jan 14 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-01-14 21:54, Andrei Alexandrescu wrote:

 I hear you but The Internet says submodules are a kind of a bummer.
Most issues I have read about is because they don't know how submodules work. It's the same thing with most tools, if you don't know how to use it the experience is going to be crappy. I'm hoping the submodules can be automatically update, i.e. on a push to some of the submodules. I'm expecting most work to be done in the submodules and the super repository will be kept in sync automatically. -- /Jacob Carlborg
Jan 14 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 15 January 2015 at 07:33:38 UTC, Jacob Carlborg 
wrote:
 On 2015-01-14 21:54, Andrei Alexandrescu wrote:

 I hear you but The Internet says submodules are a kind of a 
 bummer.
Most issues I have read about is because they don't know how submodules work. It's the same thing with most tools, if you don't know how to use it the experience is going to be crappy. I'm hoping the submodules can be automatically update, i.e. on a push to some of the submodules. I'm expecting most work to be done in the submodules and the super repository will be kept in sync automatically.
We have been using git submodules for internal dependencies of Sociomantic projects for quite a while. It is not perfect but it works good if you defines terms of usage strict enough. In this specific case I think it may work with help of few shell scripts that may reside in same top-level repo. Actually I do remember someone already trying that setup but can't find the links now. Anyone with better memory?
Jan 15 2015
parent reply "Dicebot" <public dicebot.lv> writes:
Important thing here is that starting with git 1.8.2 it is 
possible to set submodule to track head of remote branch instead 
of fixed commit hash - which fixes the maintenance issue and is 
exactly the thing needed for meta-dlang repo.
Jan 15 2015
next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Jan 15, 2015 at 04:08:14PM +0000, Dicebot via Digitalmars-d wrote:
 Important thing here is that starting with git 1.8.2 it is possible to
 set submodule to track head of remote branch instead of fixed commit
 hash - which fixes the maintenance issue and is exactly the thing
 needed for meta-dlang repo.
Very nice!! I wasn't aware of that... I must be behind the times. :-P Andrei: I think this nullifies about 50% of the naysayers' arguments against git submodules. Another issue is also that people used git submodules for things it wasn't intended for, and got burned as a result. Not really the fault of the feature. There *are* still some things to be desired, though, so it's not quite perfect yet. But it should be a lot more usable than before. T -- What are you when you run out of Monet? Baroque.
Jan 15 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/15/15 10:27 AM, H. S. Teoh via Digitalmars-d wrote:
 On Thu, Jan 15, 2015 at 04:08:14PM +0000, Dicebot via Digitalmars-d wrote:
 Important thing here is that starting with git 1.8.2 it is possible to
 set submodule to track head of remote branch instead of fixed commit
 hash - which fixes the maintenance issue and is exactly the thing
 needed for meta-dlang repo.
Very nice!! I wasn't aware of that... I must be behind the times. :-P Andrei: I think this nullifies about 50% of the naysayers' arguments against git submodules. Another issue is also that people used git submodules for things it wasn't intended for, and got burned as a result. Not really the fault of the feature. There *are* still some things to be desired, though, so it's not quite perfect yet. But it should be a lot more usable than before.
I'd be okay with reorganizing our repos as a module with the submodules dmd, druntime, phobos, and tools. For that we'd need a champion who is willing to try it, experiment with it (e.g. in a separate repo), demonstrate its advantages and discuss its disadvantages, and carry the actual porting work. It seems to me we'd need to derive a few more advantages than solving the particular matter I opened this thread with. Andrei
Jan 15 2015
parent Jacob Carlborg <doob me.com> writes:
On 2015-01-15 19:45, Andrei Alexandrescu wrote:

 I'd be okay with reorganizing our repos as a module with the submodules
 dmd, druntime, phobos, and tools. For that we'd need a champion who is
 willing to try it, experiment with it (e.g. in a separate repo),
 demonstrate its advantages and discuss its disadvantages, and carry the
 actual porting work.
I hope it will only require minimal changes in the individual submodules. Most things should be done in the super repository. There's nothing that needs to be done for a git repository to make it a submodule in another repository. -- /Jacob Carlborg
Jan 15 2015
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-01-15 17:08, Dicebot wrote:

 Important thing here is that starting with git 1.8.2 it is possible to
 set submodule to track head of remote branch instead of fixed commit
 hash - which fixes the maintenance issue and is exactly the thing needed
 for meta-dlang repo.
Yeah, but in practice I haven't noticed any difference. You still need to manually update the submodules with a command and make a commit in the super repository. I did some experiments with this at work when we moved from SVN to Git. -- /Jacob Carlborg
Jan 15 2015
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-01-14 20:50, Andrei Alexandrescu wrote:

 Currently there must be a place where druntime finds dmd for building
 purposes. Also phobos needs to find druntime. Also dlang.ord needs to
 find the source of both phobos and druntime.

 There's no specific setup of one person as much as a boring:

 /some/dir/
    dmd
    druntime
    phobos
    dlang.org
    tools

 I have a hard time making it any simpler.
Last time I tried to build the docs in dlang.org, quite a long time ago. The d-programming-language.org repository just had been moved to dlang.org. So I update the repository URL, pull the new code and tried building the docs. Nothing worked because I had forget to rename the directory. A super repository with submodules would have solved this.
 Where would that super-repo sit?
https://github.com/D-Programming-Language/dlang for example -- /Jacob Carlborg
Jan 14 2015
prev sibling parent reply "Mathias LANG" <geod24 gmail.com> writes:
On Tuesday, 13 January 2015 at 23:20:13 UTC, Andrei Alexandrescu 
wrote:
 Hey folks,


 Over the time a number of stuff has become quite duplicated 
 across our makefiles for dmd, druntime, and phobos.

 These include fetching OS and model but (newer) general-purpose 
 macros for e.g. rebasing repos, see 
 https://github.com/D-Programming-Language/dlang.org/blob/master/posix.mak#L37.

 I think it's time to reduce clutter and duplication by 
 migrating such common stuff into one common.mak file. Question 
 is, where should that file sit? One candidate is tools/ but 
 it's not really a tool. Another candidate is dmd/src/ because 
 that would be the first thing anyone depends on.

 Ideas?


 Andrei
IMO helpers (such as git helpers) are not part of the build process, so they should not be part of the build files. Few people care about make rebase. I'm not saying we should delete them (they obviously are useful to you), but moving them to tools might be a good start. For druntime / phobos build... That's really a shame we don't have a standard D tool to build D code, cross platform. If we had, not only would we promote an even saner image of our tooling ('hey look, dogfooding'), we won't have to solve the same problems over and over again, so both druntime / phobos and the tool would benefit from this, but ofc it will require some initial work. If someone wants to get this project started, I have a suggestion for the name: dub Okay, sarcasms appart, would that be a possibility for us to switch to dub in the long run ? I'm not sure dub is ready for that ATM, but I'm confident it could be made ready without much work.
Jan 14 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/14/15 8:50 AM, Mathias LANG wrote:
 IMO helpers (such as git helpers) are not part of the build process, so
 they should not be part of the build files. Few people care about make
 rebase. I'm not saying we should delete them (they obviously are useful
 to you), but moving them to tools might be a good start.
This is about scaling up and passing responsibility around. I could write a script and then tell whoever maintains dlang.org "first run that script then run make" etc. With make it turns out it's dead simple to get parallel rebase going: REBASE = git pull upstream master --rebase rebase: rebase-dlang rebase-dmd rebase-druntime rebase-phobos rebase-dlang: ; $(REBASE) rebase-dmd: ; cd $(DMD_DIR) && $(REBASE) rebase-druntime: ; cd $(DRUNTIME_DIR) && $(REBASE) rebase-phobos: ; cd $(PHOBOS_DIR) && $(REBASE) So I'm done in 6 lines and it's parallel automatically, no code needed. With zsh I'd probably go (untested)... =========================== #/bin/zsh set -e function rebase() { ( cd $1 && git pull upstream master --rebase ) } rebase . && rebase ../dmd && rebase ../druntime && rebase ../phobos && wait =========================== But wait, there's less. The script: * does not return nonzero if either subtask fails - and doing so needs e.g. temporary files and takes extra code. So I need to carefully look at the textual garbled output to make sure it all works. * does not accept setting variables DMD_DIR etc. It can be made to, but that's yet more code. * cannot run serially if I want to debug something, only parallel. It can be made do, but that's again more code! So we're looking at an ideological response "don't use make, make sucks" to a pragmatic matter - I want to get work done, and we shouldn't use inferior/more intensive technical solutions on an ideological basis alone. I agree make sucks! But that doesn't suddenly eliminate my common sense.
 For druntime / phobos build... That's really a shame we don't have a
 standard D tool to build D code, cross platform.
 If we had, not only would we promote an even saner image of our tooling
 ('hey look, dogfooding'), we won't have to solve the same problems over
 and over again, so both druntime / phobos and the tool would benefit
 from this, but ofc it will require some initial work.
 If someone wants to get this project started, I have a suggestion for
 the name: dub

 Okay, sarcasms appart, would that be a possibility for us to switch to
 dub in the long run ? I'm not sure dub is ready for that ATM, but I'm
 confident it could be made ready without much work.
I'd be glad to switch to dub but I confess at this time I can't invest the time. I'm already spread too thin. Andrei
Jan 14 2015
parent reply "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Wednesday, 14 January 2015 at 20:07:12 UTC, Andrei 
Alexandrescu wrote:
 On 1/14/15 8:50 AM, Mathias LANG wrote:
 IMO helpers (such as git helpers) are not part of the build 
 process, so
 they should not be part of the build files. Few people care 
 about make
 rebase. I'm not saying we should delete them (they obviously 
 are useful
 to you), but moving them to tools might be a good start.
This is about scaling up and passing responsibility around. I could write a script and then tell whoever maintains dlang.org "first run that script then run make" etc. With make it turns out it's dead simple to get parallel rebase going: REBASE = git pull upstream master --rebase rebase: rebase-dlang rebase-dmd rebase-druntime rebase-phobos rebase-dlang: ; $(REBASE) rebase-dmd: ; cd $(DMD_DIR) && $(REBASE) rebase-druntime: ; cd $(DRUNTIME_DIR) && $(REBASE) rebase-phobos: ; cd $(PHOBOS_DIR) && $(REBASE) So I'm done in 6 lines and it's parallel automatically, no code needed. With zsh I'd probably go (untested)... =========================== #/bin/zsh set -e function rebase() { ( cd $1 && git pull upstream master --rebase ) } rebase . && rebase ../dmd && rebase ../druntime && rebase ../phobos && wait =========================== But wait, there's less. The script: * does not return nonzero if either subtask fails - and doing so needs e.g. temporary files and takes extra code. So I need to carefully look at the textual garbled output to make sure it all works. * does not accept setting variables DMD_DIR etc. It can be made to, but that's yet more code. * cannot run serially if I want to debug something, only parallel. It can be made do, but that's again more code! So we're looking at an ideological response "don't use make, make sucks" to a pragmatic matter - I want to get work done, and we shouldn't use inferior/more intensive technical solutions on an ideological basis alone. I agree make sucks! But that doesn't suddenly eliminate my common sense.
 For druntime / phobos build... That's really a shame we don't 
 have a
 standard D tool to build D code, cross platform.
 If we had, not only would we promote an even saner image of 
 our tooling
 ('hey look, dogfooding'), we won't have to solve the same 
 problems over
 and over again, so both druntime / phobos and the tool would 
 benefit
 from this, but ofc it will require some initial work.
 If someone wants to get this project started, I have a 
 suggestion for
 the name: dub

 Okay, sarcasms appart, would that be a possibility for us to 
 switch to
 dub in the long run ? I'm not sure dub is ready for that ATM, 
 but I'm
 confident it could be made ready without much work.
I'd be glad to switch to dub but I confess at this time I can't invest the time. I'm already spread too thin. Andrei
So a dub file for phobos is needed? And maybe druntime too? I'll see what I can do.
Jan 14 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/14/15 3:14 PM, Rikki Cattermole wrote:
 So a dub file for phobos is needed? And maybe druntime too? I'll see
 what I can do.
That would be very interesting, thanks. Prolly druntime is easiest - one less dependency. Please share all discoveries you make with the core team here. -- Andrei
Jan 14 2015
next sibling parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 15/01/2015 12:24 p.m., Andrei Alexandrescu wrote:
 On 1/14/15 3:14 PM, Rikki Cattermole wrote:
 So a dub file for phobos is needed? And maybe druntime too? I'll see
 what I can do.
That would be very interesting, thanks. Prolly druntime is easiest - one less dependency. Please share all discoveries you make with the core team here. -- Andrei
Phobos is done (2.066 branch). I'll submit PR when soon. Oh there will also be atleast two fixes. Somebody forgot to add some version statements in linux/osx only code and since I'm lazy no way I'm specifying every single file to dub.
Jan 14 2015
parent reply "Mathias LANG" <geod24 gmail.com> writes:
On Wednesday, 14 January 2015 at 23:58:45 UTC, Rikki Cattermole 
wrote:
 On 15/01/2015 12:24 p.m., Andrei Alexandrescu wrote:
 On 1/14/15 3:14 PM, Rikki Cattermole wrote:
 So a dub file for phobos is needed? And maybe druntime too? 
 I'll see
 what I can do.
That would be very interesting, thanks. Prolly druntime is easiest - one less dependency. Please share all discoveries you make with the core team here. -- Andrei
Phobos is done (2.066 branch). I'll submit PR when soon. Oh there will also be atleast two fixes. Somebody forgot to add some version statements in linux/osx only code and since I'm lazy no way I'm specifying every single file to dub.
excludedSourceFiles accept patterns and platform suffixes.
Jan 14 2015
parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 15/01/2015 1:15 p.m., Mathias LANG wrote:
 On Wednesday, 14 January 2015 at 23:58:45 UTC, Rikki Cattermole wrote:
 On 15/01/2015 12:24 p.m., Andrei Alexandrescu wrote:
 On 1/14/15 3:14 PM, Rikki Cattermole wrote:
 So a dub file for phobos is needed? And maybe druntime too? I'll see
 what I can do.
That would be very interesting, thanks. Prolly druntime is easiest - one less dependency. Please share all discoveries you make with the core team here. -- Andrei
Phobos is done (2.066 branch). I'll submit PR when soon. Oh there will also be atleast two fixes. Somebody forgot to add some version statements in linux/osx only code and since I'm lazy no way I'm specifying every single file to dub.
excludedSourceFiles accept patterns and platform suffixes.
I'm using it for std.metastrings. Not really a loss since the distributed version doesn't include it (also doesn't compile). In the case of the version statements? Its two line additions in total. They definitely should be there considering.
Jan 14 2015
prev sibling parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 15/01/2015 12:24 p.m., Andrei Alexandrescu wrote:
 On 1/14/15 3:14 PM, Rikki Cattermole wrote:
 So a dub file for phobos is needed? And maybe druntime too? I'll see
 what I can do.
That would be very interesting, thanks. Prolly druntime is easiest - one less dependency. Please share all discoveries you make with the core team here. -- Andrei
Druntime and phobos now can be built and even more importantly you can have druntime as a dependency to phobos. In fact it is. Phobos supports being built as a dynamic library (there is a configuration for for it). So if you want to switch to dynamic library at any time its as simple as adding: "subConfigurations": { "phobos": "dynamicLibrary" } I'm pretty sure anyway. Because of the current state of assembly files, we really need to sort it out. I understand that not all things are really possible in an inline assembler. But at the very least we should be able to compile the damn thing. Dub can run external processes but its best to leave it up to the end dev to run the c/assembly generation. When they want it regenerated. I think its best we commit most of the binary outputs. Mostly c/assembly generation. Not the D stuff. I've added information to the README on how to build based upon dub. Docs so far have not been included. But that adds one more repo I need to get my hands dirty with.
Jan 14 2015
next sibling parent reply "Mathias LANG" <geod24 gmail.com> writes:
On Thursday, 15 January 2015 at 01:48:29 UTC, Rikki Cattermole 
wrote:
 Druntime and phobos now can be built and even more importantly 
 you can have druntime as a dependency to phobos. In fact it is.
 Phobos supports being built as a dynamic library (there is a 
 configuration for for it).
 So if you want to switch to dynamic library at any time its as 
 simple as adding:
 "subConfigurations": {
 	"phobos": "dynamicLibrary"
 }
 I'm pretty sure anyway.
Thanks ! Last time I checked, building dynamic library with dub was buggy. In addition, we can't have druntime has a dependency to phobos with dynamic library (yet). See https://github.com/D-Programming-Language/dub/blob/master/source/dub/generators/generator.d#L152 But having the static version in is a good start :)
Jan 14 2015
parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 15/01/2015 2:58 p.m., Mathias LANG wrote:
 On Thursday, 15 January 2015 at 01:48:29 UTC, Rikki Cattermole wrote:
 Druntime and phobos now can be built and even more importantly you can
 have druntime as a dependency to phobos. In fact it is.
 Phobos supports being built as a dynamic library (there is a
 configuration for for it).
 So if you want to switch to dynamic library at any time its as simple
 as adding:
 "subConfigurations": {
     "phobos": "dynamicLibrary"
 }
 I'm pretty sure anyway.
Thanks ! Last time I checked, building dynamic library with dub was buggy. In addition, we can't have druntime has a dependency to phobos with dynamic library (yet). See https://github.com/D-Programming-Language/dub/blob/master/source/dub/generators/generator.d#L152 But having the static version in is a good start :)
The configuration is there. Atleast I'm prepared! Anyway static version is default, so as long as dynamic isn't advertised too much we're good.
Jan 14 2015
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-01-15 02:48, Rikki Cattermole wrote:

 Dub can run external processes but its best to leave it up to the end
 dev to run the c/assembly generation. When they want it regenerated.
 I think its best we commit most of the binary outputs.
 Mostly c/assembly generation. Not the D stuff.
Why don't you want to build the C/assembly stuff? The user is already expected to have a C compiler installed. -- /Jacob Carlborg
Jan 14 2015
parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 15/01/2015 8:35 p.m., Jacob Carlborg wrote:
 On 2015-01-15 02:48, Rikki Cattermole wrote:

 Dub can run external processes but its best to leave it up to the end
 dev to run the c/assembly generation. When they want it regenerated.
 I think its best we commit most of the binary outputs.
 Mostly c/assembly generation. Not the D stuff.
Why don't you want to build the C/assembly stuff? The user is already expected to have a C compiler installed.
Imagine your wanting to get some new flash feature (doesn't require compiler change) by just adding druntime/phobos as a dependency via dub. You don't really want to recompile the assembly/c stuff that almost definitely hasn't changed. Not many people may want to do that. But it should be reasonable to do so. But this is mostly a Windows issue then posix. There its pretty well defined. Windows ergg.
Jan 15 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-01-15 09:30, Rikki Cattermole wrote:

 Imagine your wanting to get some new flash feature (doesn't require
 compiler change) by just adding druntime/phobos as a dependency via dub.
 You don't really want to recompile the assembly/c stuff that almost
 definitely hasn't changed.
Why not? It's more likely you'll run in to problems with forgetting to compile it when it really has changed.
 Not many people may want to do that. But it should be reasonable to do so.
 But this is mostly a Windows issue then posix. There its pretty well
 defined. Windows ergg.
If it's only a problem on Windows, don't compile it on Windows :). The other platforms should suffer because of this. -- /Jacob Carlborg
Jan 15 2015
parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 16/01/2015 1:19 a.m., Jacob Carlborg wrote:
 On 2015-01-15 09:30, Rikki Cattermole wrote:

 Imagine your wanting to get some new flash feature (doesn't require
 compiler change) by just adding druntime/phobos as a dependency via dub.
 You don't really want to recompile the assembly/c stuff that almost
 definitely hasn't changed.
Why not? It's more likely you'll run in to problems with forgetting to compile it when it really has changed.
In this case I think its pretty close between checking in and not. Atleast for me, I generally keep my D environment separate from e.g. VC so things like this could get difficult unless it was configured per machine like it is for dmd.
 Not many people may want to do that. But it should be reasonable to do
 so.
 But this is mostly a Windows issue then posix. There its pretty well
 defined. Windows ergg.
If it's only a problem on Windows, don't compile it on Windows :). The other platforms should suffer because of this.
Technically there is one assembly file required by druntime for Windows that can't even be compiled on any 32 or 64bit computer... and is checked into git.
Jan 15 2015