www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - D, SCons, Dub

reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
Just in case anyone gives a ####:

I have submitted a pull request that adds ProgramAllAtOnce as a builder
in the dmd, ldc, and gdc tools of SCons. This does an 'all at once'
compilation in a single compiler instantiation, unlike the standard
module at a time compilation and then link the program. There are lots
of arguments about whether "all at once" is at all useful, I have added
it simply because it is the only way Unit-Threaded works.

If you want to use it prior to it being accepted and released, then
feel free to use the tools in the repository https://github.com/russel/
SCons_D_Experiment

Also in there you will spot a new tool dub. I am now using this to grab
unit-threaded from the Dub repository in a SCons build. See for example
 https://github.com/russel/ApproxGC

--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Apr 17
next sibling parent reply =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Monday, 17 April 2017 at 17:51:33 UTC, Russel Winder wrote:
 Just in case anyone gives a ####:

 I have submitted a pull request that adds ProgramAllAtOnce as a 
 builder in the dmd, ldc, and gdc tools of SCons. This does an 
 'all at once' compilation in a single compiler instantiation, 
 unlike the standard module at a time compilation and then link 
 the program. There are lots of arguments about whether "all at 
 once" is at all useful, I have added it simply because it is 
 the only way Unit-Threaded works.
Nice. I'm using SCons in my everyday development. I'll give your feature a try. Meson has recently added a similar feature. Meson calls it a "unity build".
 If you want to use it prior to it being accepted and released, 
 then feel free to use the tools in the repository 
 https://github.com/russel/ SCons_D_Experiment

 Also in there you will spot a new tool dub. I am now using this 
 to grab
 unit-threaded from the Dub repository in a SCons build. See for 
 example
  https://github.com/russel/ApproxGC
Cool. Thanks.
Apr 17
parent Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 2017-04-17 at 18:53 +0000, Nordl=C3=B6w via Digitalmars-d wrote:
=20
[=E2=80=A6]
 Nice. I'm using SCons in my everyday development. I'll give your=C2=A0
 feature a try.
My pull request is currently being reviewed and some updates are needed. These will appear immediately in the SCons_D_Experiment. I'll post once the pull request gets merged into SCons mainline. Then it just awaits a release, unless you use default/tip from the mainline repository.
 Meson has recently added a similar feature. Meson calls it a=C2=A0
 "unity build".
Is there documentation? I need this immediately as unit-threaded only works with whole source compilation.
 If you want to use it prior to it being accepted and released,=C2=A0
 then feel free to use the tools in the repository=C2=A0
 https://github.com/russel/ SCons_D_Experiment
=20
 Also in there you will spot a new tool dub. I am now using this=C2=A0
 to grab
 unit-threaded from the Dub repository in a SCons build. See for=C2=A0
 example
 =C2=A0https://github.com/russel/ApproxGC
=20 Cool. =20
:-) --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Apr 17
prev sibling next sibling parent reply Alex <alexibu.nospam me.com> writes:
On Monday, 17 April 2017 at 17:51:33 UTC, Russel Winder wrote:
 Just in case anyone gives a ####:

 I have submitted a pull request that adds ProgramAllAtOnce
I have been trying to do the opposite with scons - incremental builds. In c++ the .h and .cpp files allowed the compilation from many edits to be limited to one translation unit. One thing people are often confused about is whether everything needs to be in the header (or di file). An influence is that most of the code they see written by skilled language users is libraries like the standard library. These typically benefit little from interfaces files because everything is templates and must be in the header to compile. Hence why boost is mostly headers in c++ land and the D ecosystem encourages all at once / all everytime compilation. In a large code base this becomes cumbersome. Changing a unit test should not result in everything recompiling. I think there is a difference between library style code and application code. My application code doesn't have templates and can make use of di files. It makes sense that the language developers spend most of their time on library type code, and have been focused on it. I just need to work out how to get scons to recognise two targets from one builder properly : the object file and the di file. I can add DFLAGs to make a Object builder make the di files too, and the d scanner finds di files where d files are not available, but the build tree dependency from d to di file is missing. Dies anyone know how to do this ?
Apr 17
next sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Monday, 17 April 2017 at 20:56:55 UTC, Alex wrote:
 On Monday, 17 April 2017 at 17:51:33 UTC, Russel Winder wrote:
 Just in case anyone gives a ####:

 I have submitted a pull request that adds ProgramAllAtOnce
I have been trying to do the opposite with scons - incremental builds. In c++ the .h and .cpp files allowed the compilation from many edits to be limited to one translation unit.
In D, this is nearly always slower (I measured). Compiling per package, however, is a lot faster. Atila
Apr 17
parent Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 2017-04-17 at 22:59 +0000, Atila Neves via Digitalmars-d wrote:
 [=E2=80=A6]
 In D, this is nearly always slower (I measured). Compiling per=C2=A0
 package, however, is a lot faster.
I have only ever needed file-by-file or whole source compilation, I have never needed package-oriented compilation. It sounds like this needs adding to SCons and Meson. What is needed is a really small project showing the idea so that it can be used as a test case. --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Apr 17
prev sibling parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 2017-04-17 at 20:56 +0000, Alex via Digitalmars-d wrote:
 On Monday, 17 April 2017 at 17:51:33 UTC, Russel Winder wrote:
 Just in case anyone gives a ####:
=20
 I have submitted a pull request that adds ProgramAllAtOnce
=20 I have been trying to do the opposite with scons - incremental=C2=A0 builds. In c++ the .h and .cpp files allowed the compilation from=C2=A0 many edits to be limited to one translation unit.
Per module compilation then linking is the default for the Program builder in SCons, that is already there and has been for over a decade.
 One thing people are often confused about is whether everything=C2=A0
 needs to be in the header (or di file).
Uuurrr=E2=80=A6 there is no separation of header and code in D, just module= s. And packages. SCons currently has no "per package" builder. I guess I need to add this.
 An influence is that most of the code they see written by skilled=C2=A0
 language users is libraries like the standard library. These=C2=A0
 typically benefit little from interfaces files because everything=C2=A0
 is templates and must be in the header to compile. Hence why=C2=A0
 boost is mostly headers in c++ land and the D ecosystem=C2=A0
 encourages all at once / all everytime compilation.
=20
 In a large code base this becomes cumbersome. Changing a unit=C2=A0
 test should not result in everything recompiling.
=20
 I think there is a difference between library style code and=C2=A0
 application code. My application code doesn't have templates and=C2=A0
 can make use of di files. It makes sense that the language=C2=A0
 developers spend most of their time on library type code, and=C2=A0
 have been focused on it.
=20
 I just need to work out how to get scons to recognise two targets=C2=A0
 from one builder properly : the object file and the di file.
 I can add DFLAGs to make a Object builder make the di files too,=C2=A0
 and the d scanner finds di files where d files are not available,=C2=A0
 but the build tree dependency from d to di file is missing. Dies=C2=A0
 anyone know how to do this ?
Currently there is no explicit handling of di files in SCons. If you can come up with a small project that exhibits the problem you need solved, I can add it as a test case in the SCons system thereby creating a need for a fix. (Tests in SCons for the tools are end-to-end tests, so need a project exhibiting a problem.) =20 --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Apr 17
parent reply Alex <alexibu.nospam me.com> writes:
On Tuesday, 18 April 2017 at 06:51:51 UTC, Russel Winder wrote:
 I have been trying to do the opposite with scons - incremental 
 builds. In c++ the .h and .cpp files allowed the compilation 
 from many edits to be limited to one translation unit.
Per module compilation then linking is the default for the Program builder in SCons, that is already there and has been for over a decade.
I know, I have been using your work in SCons to build in this mode since it was first available.
 One thing people are often confused about is whether 
 everything needs to be in the header (or di file).
Uuurrr… there is no separation of header and code in D, just modules. And packages. SCons currently has no "per package" builder. I guess I need to add this.
I realise there are no headers, that is the problem. SCons can't tell whether a change to a module affects the interface, only affects the implementation or only affects a unit test. This means everything that imports a module is recompiled on every edit, even if only the unit test was modified, or a formatting change was made. I am not suggesting the D way is bad compared to C++, I am suggesting we can have the best of both worlds by automatically generating 'headers' or .di files from modules. Then having dependent modules read only the .di file instead of the .d file. This way SCons won't recompile dependencies unless necessary. This is the main purpose of using SCons (perfect incremental builds) which is largely defeated currently by D.
 Currently there is no explicit handling of di files in SCons.
I know, I have been working through the guts of SCons for (a little bit too long to mention) trying to make it both output two products with proper dependencies (.o and .di) and to find the di files correctly. SCons seems to have an ability to find the original .d files in the project even if the compiler can't.
 If you can come up with a small project that exhibits the 
 problem you need solved, I can add it as a test case in the 
 SCons system thereby creating a need for a fix.
Thanks for that kind offer Russel, I will make a minimal example project.
 (Tests in SCons for the tools are end-to-end tests, so need a 
 project
 exhibiting a problem.)
OK.
Apr 18
next sibling parent Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 2017-04-18 at 07:34 +0000, Alex via Digitalmars-d wrote:
=20
[=E2=80=A6]
 I know, I have been using your work in SCons to build in this=C2=A0
 mode since it was first available.
Excellent. :-) [=E2=80=A6]
=20
I realise there are no headers, that is the problem. SCons can't=C2=A0 tell whether a change to a module affects the interface, only=C2=A0 affects the implementation or only affects a unit test. This=C2=A0 means everything that imports a module is recompiled on every=C2=A0 edit, even if only the unit test was modified, or a formatting=C2=A0 change was made.
SCons should be immune to whitespace changes, but actual reformatting is likely to be a change. Hummm=E2=80=A6 it is not clear in my mind just now whether that dependency = is build into the SCons build DAG.=C2=A0Whilst the C and C++ builders go to great lengths to deal with the #include dependency tree, I am not sure the D tools parse the D source import dependency tree. It probably should, but just now =E2=80=93 I believe =E2=80=93 it is just module by mod= ule compilation. If there are API changes then SCons will not follow that through for D codes.=20 =20
 I am not suggesting the D way is bad compared to C++, I am=C2=A0
 suggesting we can have the best of both worlds by automatically=C2=A0
 generating 'headers' or .di files from modules. Then having=C2=A0
 dependent modules read only the .di file instead of the .d file.
 This way SCons won't recompile dependencies unless necessary.=C2=A0
 This is the main purpose of using SCons (perfect incremental=C2=A0
 builds) which is largely defeated currently by D.
I have never trodden the .di file path and so the SCons tools do not generate them. A flag could easily be added so that an emitter of the builder could generate the .di as well as the .o file. The tools are not set up for that as yet, but it could be done. Being sent small projects as test cases to drive adding features is likely the best route. Feel free to send in issues on https://github.c om/russel/SCons_D_Experiment [=E2=80=A6]
 I know, I have been working through the guts of SCons for (a=C2=A0
 little bit too long to mention)=C2=A0=C2=A0trying to make it both output=
=C2=A0
 two products with proper dependencies (.o and .di) and to find=C2=A0
 the di files correctly. SCons seems to have an ability to find=C2=A0
 the original .d files in the project even if the compiler can't.
It maybe that the missing link here is that the D builders currently do not have emitters. With an emitter added doing the .di as well as .o files should be very straightforward =E2=80=93 but maybe not easy. [=E2=80=A6] --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Apr 18
prev sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Tue, Apr 18, 2017 at 07:34:34AM +0000, Alex via Digitalmars-d wrote:
[...]
 I realise there are no headers, that is the problem. SCons can't tell
 whether a change to a module affects the interface, only affects the
 implementation or only affects a unit test. This means everything that
 imports a module is recompiled on every edit, even if only the unit
 test was modified, or a formatting change was made.
This is not a problem if you use env.Decider('content'), in which case the worst that will happen is that the module with the whitespace / unittest change will be recompiled, but everything else that depends on that module won't be touched because the object file will have the same content as before.
 I am not suggesting the D way is bad compared to C++, I am suggesting
 we can have the best of both worlds by automatically generating
 'headers' or .di files from modules. Then having dependent modules
 read only the .di file instead of the .d file.
[...] This is not workable for D code that uses a lot of templates. In my own projects I never use .di files because it suffices to import the .d file directly. In the D world, generally the only time you'd use .di files is (1) interfacing with proprietary libraries, or (2) interfacing with C/C++ code (but even in this latter case, it's common practice to simply declare C/C++ functions in extern(C) or extern(C++) blocks inside a .d file, rather than have a .di file). It's pretty unnatural to use .di files anywhere else in the typical D project. T -- "I'm running Windows '98." "Yes." "My computer isn't working now." "Yes, you already said that." -- User-Friendly
Apr 18
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Monday, 17 April 2017 at 17:51:33 UTC, Russel Winder wrote:
 Just in case anyone gives a ####:

 I have submitted a pull request that adds ProgramAllAtOnce as a 
 builder in the dmd, ldc, and gdc tools of SCons. This does an 
 'all at once' compilation in a single compiler instantiation, 
 unlike the standard module at a time compilation and then link 
 the program. There are lots of arguments about whether "all at 
 once" is at all useful, I have added it simply because it is 
 the only way Unit-Threaded works.
It's the only way __traits(getUnitTests) works. Unfortunately. I'm just going to fix that dmd bug myself. Atila
Apr 17
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/17/17 6:58 PM, Atila Neves wrote:
 On Monday, 17 April 2017 at 17:51:33 UTC, Russel Winder wrote:
 Just in case anyone gives a ####:

 I have submitted a pull request that adds ProgramAllAtOnce as a 
 builder in the dmd, ldc, and gdc tools of SCons. This does an 'all at 
 once' compilation in a single compiler instantiation, unlike the 
 standard module at a time compilation and then link the program. There 
 are lots of arguments about whether "all at once" is at all useful, I 
 have added it simply because it is the only way Unit-Threaded works.
It's the only way __traits(getUnitTests) works. Unfortunately. I'm just going to fix that dmd bug myself.
Thanks!! -- Andrei
Apr 17