www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DMD test suite not runnable on Debian/Linux

reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
I'm trying to track down an autotester failure in a DMD PR, and getting
very frustrated that the DMD test suite does not run on Debian/Linux
because of the whole PIE-by-default fiasco that necessitates compiling
everything with -fPIC.

(1) Currently, there is no way to customize how test/Makefile compiles
d_do_tests.d, so there is no way to insert -fPIC into the offending
compile line.

(2) d_do_tests itself runs compile commands with `-conf=`, so putting
-fPIC inside dmd.conf has no effect (where otherwise it does the right
thing when dmd is invoked "normally"). There's no obvious way to make
customize the compile command used for each test: I tried setting the
environment variable ARGS but that just messed up the test runner's
expected outputs. Tried to add -fPIC to the DMD environment variable but
that blew up in a different way.

So basically, I'm stuck up PIE creek without a paddle, and have no way
to locally test DMD changes without hogging more load on the autotester.

Why is something so simple so hard to do?!


T

-- 
Жил-был король когда-то, при нём блоха жила.
Jun 09
next sibling parent reply Seb <seb wilzba.ch> writes:
On Friday, 9 June 2017 at 22:14:30 UTC, H. S. Teoh wrote:
 I'm trying to track down an autotester failure in a DMD PR, and 
 getting very frustrated that the DMD test suite does not run on 
 Debian/Linux because of the whole PIE-by-default fiasco that 
 necessitates compiling everything with -fPIC.

 (1) Currently, there is no way to customize how test/Makefile 
 compiles d_do_tests.d, so there is no way to insert -fPIC into 
 the offending compile line.
Once this PR is merged, it will at least use the default conf file: https://github.com/dlang/dmd/pull/6870 (the config file is now in generated and this lead to nice errors on my machine)
 (2) d_do_tests itself runs compile commands with `-conf=`, so 
 putting -fPIC inside dmd.conf has no effect (where otherwise it 
 does the right thing when dmd is invoked "normally"). There's 
 no obvious way to make customize the compile command used for 
 each test: I tried setting the environment variable ARGS but 
 that just messed up the test runner's expected outputs. Tried 
 to add -fPIC to the DMD environment variable but that blew up 
 in a different way.
Do you mean DFLAGS with DMD environment variable?
 So basically, I'm stuck up PIE creek without a paddle, and have 
 no way to locally test DMD changes without hogging more load on 
 the autotester.
Don't worry about that. It's just a computer. As a temporary workaround, docker might be worth considering.
 Why is something so simple so hard to do?!
I think it's simply because not many people use the DMD test suite and those who do are used to the pain or have their custom-built tricks. Also I guess that there are two groups of people: A) wow that's complicated and frustrating -> move on to the next great thing in life B) eh what the heck. I have seen worse... (And not many in-between).
Jun 09
next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Fri, Jun 09, 2017 at 10:34:00PM +0000, Seb via Digitalmars-d wrote:
 On Friday, 9 June 2017 at 22:14:30 UTC, H. S. Teoh wrote:
 I'm trying to track down an autotester failure in a DMD PR, and
 getting very frustrated that the DMD test suite does not run on
 Debian/Linux because of the whole PIE-by-default fiasco that
 necessitates compiling everything with -fPIC.
 
 (1) Currently, there is no way to customize how test/Makefile
 compiles d_do_tests.d, so there is no way to insert -fPIC into the
 offending compile line.
Once this PR is merged, it will at least use the default conf file: https://github.com/dlang/dmd/pull/6870 (the config file is now in generated and this lead to nice errors on my machine)
But you see, that's the problem. What happens if the auto-generated config file needs further adjustment?
 (2) d_do_tests itself runs compile commands with `-conf=`, so
 putting -fPIC inside dmd.conf has no effect (where otherwise it does
 the right thing when dmd is invoked "normally"). There's no obvious
 way to make customize the compile command used for each test: I
 tried setting the environment variable ARGS but that just messed up
 the test runner's expected outputs. Tried to add -fPIC to the DMD
 environment variable but that blew up in a different way.
Do you mean DFLAGS with DMD environment variable?
There is no DFLAGS defined in d_do_tests.d. The only configurable parameter I can see is the DMD variable, so I tried to bluff d_do_tests by setting DMD="dmd -fPIC", but that also didn't work, maybe because it doesn't go through shell interpolation so it tries to find a non-existent executable named "dmd -fPIC". Anyway, I eventually went back to my trusty old anti-makefile workaround: create a wrapper shell script that does this: #!/bin/sh /path/to/dmd -fPIC -I/path/to/druntime/import -I/path/to/phobos "$ " (Yes, the -I's are necessary, I don't know why and don't really want to at this point.) Save this to /path/to/dmdwrapper, and then run make with DMD=/path/to/dmdwrapper, and lo and behold, finally the miserable test suite is runnable. That was 2 hours' worth of frustration just to get the testsuite running. I dunno about you, but being unable to run the dmd testsuite seems to me to be a big turnoff for potential dmd contributors. I surmise not every potential dmd contributor is as persistent as I am. :-D
 So basically, I'm stuck up PIE creek without a paddle, and have no
 way to locally test DMD changes without hogging more load on the
 autotester.
Don't worry about that. It's just a computer. As a temporary workaround, docker might be worth considering.
Wow. Way to bring out the shotgun to kill an ant! I think installing docker would be total overkill for something that really ought to work out-of-the-box...!
 Why is something so simple so hard to do?!
I think it's simply because not many people use the DMD test suite and those who do are used to the pain or have their custom-built tricks. Also I guess that there are two groups of people: A) wow that's complicated and frustrating -> move on to the next great thing in life B) eh what the heck. I have seen worse... (And not many in-between).
Ha, so now I've invented my own custom-built trick to make it work, and so that's no help to the next would-be dmd contributor that comes along and can't figure out how to make it work. :-/ // On a tangential note, now that I finally got the miserable test suite running, I've managed to figure out that the problem in my dmd PR was caused by src/ddmd/globals.h not being auto-generated from src/ddmd/globals.d, so when I added new fields to the Global struct, the C++ code that use globals.h with the old definition crashed. I suppose this is just a relic from the pre-self-hosting days, but it would be nice if these last .h files were gotten rid of in the near future. That, or we should (ab)use the .di generation facility or some sed script to generate these .h files, so that they won't go out-of-sync inadvertently. T -- LINUX = Lousy Interface for Nefarious Unix Xenophobes.
Jun 09
parent reply ketmar <ketmar ketmar.no-ip.org> writes:
H. S. Teoh wrote:

 On a tangential note, now that I finally got the miserable test suite
 running, I've managed to figure out that the problem in my dmd PR was
 caused by src/ddmd/globals.h not being auto-generated from
 src/ddmd/globals.d, so when I added new fields to the Global struct, the
 C++ code that use globals.h with the old definition crashed. I suppose
 this is just a relic from the pre-self-hosting days, but it would be
 nice if these last .h files were gotten rid of in the near future.

 That, or we should (ab)use the .di generation facility or some sed
 script to generate these .h files, so that they won't go out-of-sync
 inadvertently.
we can use UDAs to do that! just make the generator import the relevand dmd parts, and then use compile-time introspection to build tables, so generator can just write 'em to file in runtime. ;-)
Jun 09
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Sat, Jun 10, 2017 at 02:27:13AM +0300, ketmar via Digitalmars-d wrote:
 H. S. Teoh wrote:
 
 On a tangential note, now that I finally got the miserable test
 suite running, I've managed to figure out that the problem in my dmd
 PR was caused by src/ddmd/globals.h not being auto-generated from
 src/ddmd/globals.d, so when I added new fields to the Global struct,
 the C++ code that use globals.h with the old definition crashed. I
 suppose this is just a relic from the pre-self-hosting days, but it
 would be nice if these last .h files were gotten rid of in the near
 future.
 
 That, or we should (ab)use the .di generation facility or some sed
 script to generate these .h files, so that they won't go out-of-sync
 inadvertently.
we can use UDAs to do that! just make the generator import the relevand dmd parts, and then use compile-time introspection to build tables, so generator can just write 'em to file in runtime. ;-)
Sure! But ... uhm... what has UDAs gotta do with that? A straight `foreach (memb; __traits(allMembers, T))` should do the trick. T -- A mathematician is a device for turning coffee into theorems. -- P. Erdos
Jun 09
next sibling parent reply Seb <seb wilzba.ch> writes:
On Saturday, 10 June 2017 at 00:16:13 UTC, H. S. Teoh wrote:
 On Sat, Jun 10, 2017 at 02:27:13AM +0300, ketmar via 
 Digitalmars-d wrote:
 H. S. Teoh wrote:
 
 [...]
we can use UDAs to do that! just make the generator import the relevand dmd parts, and then use compile-time introspection to build tables, so generator can just write 'em to file in runtime. ;-)
Sure! But ... uhm... what has UDAs gotta do with that? A straight `foreach (memb; __traits(allMembers, T))` should do the trick. T
Are you aware of this excellent PR? https://github.com/dlang/dmd/pull/5082
Jun 09
parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Sat, Jun 10, 2017 at 12:33:39AM +0000, Seb via Digitalmars-d wrote:
 On Saturday, 10 June 2017 at 00:16:13 UTC, H. S. Teoh wrote:
 On Sat, Jun 10, 2017 at 02:27:13AM +0300, ketmar via Digitalmars-d
 wrote:
 H. S. Teoh wrote:
 
 [...]
we can use UDAs to do that! just make the generator import the relevand dmd parts, and then use compile-time introspection to build tables, so generator can just write 'em to file in runtime. ;-)
Sure! But ... uhm... what has UDAs gotta do with that? A straight `foreach (memb; __traits(allMembers, T))` should do the trick. T
Are you aware of this excellent PR? https://github.com/dlang/dmd/pull/5082
Whoa. This is exactly what I've been looking for! T -- The best way to destroy a cause is to defend it poorly.
Jun 09
prev sibling parent ketmar <ketmar ketmar.no-ip.org> writes:
H. S. Teoh wrote:

 On Sat, Jun 10, 2017 at 02:27:13AM +0300, ketmar via Digitalmars-d wrote:
 H. S. Teoh wrote:
 
 On a tangential note, now that I finally got the miserable test
 suite running, I've managed to figure out that the problem in my dmd
 PR was caused by src/ddmd/globals.h not being auto-generated from
 src/ddmd/globals.d, so when I added new fields to the Global struct,
 the C++ code that use globals.h with the old definition crashed. I
 suppose this is just a relic from the pre-self-hosting days, but it
 would be nice if these last .h files were gotten rid of in the near
 future.
 That, or we should (ab)use the .di generation facility or some sed
 script to generate these .h files, so that they won't go out-of-sync
 inadvertently.
we can use UDAs to do that! just make the generator import the relevand dmd parts, and then use compile-time introspection to build tables, so generator can just write 'em to file in runtime. ;-)
Sure! But ... uhm... what has UDAs gotta do with that? A straight `foreach (memb; __traits(allMembers, T))` should do the trick.
i thought about situations where you may need to exclude something from generated header... but this surely has no sense, as one cannot exclude fields or virtual methods from generated classes, so `extern(C++)` is surely enough. well, i like to overengineer! ;-)
Jun 09
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2017-06-10 00:34, Seb wrote:

 Don't worry about that. It's just a computer. As a temporary workaround,
 docker might be worth considering.
Wow. Just modify the Makefile. -- /Jacob Carlborg
Jun 10
prev sibling parent ketmar <ketmar ketmar.no-ip.org> writes:
it's even worser than that: unittests ultimatively fails with binutils 2.28 
(at least on x86 without PIC), so i was forced to downgrade binutils to 2.26.
Jun 09