www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.ldc - Large number of tests failing - 64bit LDC with MSVC - Append Operator

reply "Kevin Brogan" <kevin brogan.ca> writes:
I've been able to successfully build LDC on 64 bit windows using 
msvc, but I've noticed that a large number of test suites fail 
with a segmentation fault instead of a regular error result.

I've been able to determine that many of the segmentation faults 
is due to the append operator.

The fault only occurs when using a variable, and only when two 
append operators are used in the same statement.

I was wondering if anyone can replicate this in their environment 
or have an idea of how to determine the cause.

An example is here:

import std.c.stdio : printf;
void main()
{
     string variable = "variable";
     printf("made a string\n");

     string result;

     result = "before " ~ variable;
     printf("before\n");

     result = variable ~ " after";
     printf("after\n");

     result = "before " ~ "novar" ~ " after";
     printf("novar\n");

     result = "before " ~ variable ~ " after"; // this line will 
seg fault
     printf("crashes before getting here\n");
}
Feb 24 2015
next sibling parent "Kai Nacke" <kai redstar.de> writes:
Hi Kevin!

On Wednesday, 25 February 2015 at 00:02:33 UTC, Kevin Brogan 
wrote:
 I've been able to successfully build LDC on 64 bit windows 
 using msvc, but I've noticed that a large number of test suites 
 fail with a segmentation fault instead of a regular error 
 result.

 I've been able to determine that many of the segmentation 
 faults is due to the append operator.

 The fault only occurs when using a variable, and only when two 
 append operators are used in the same statement.

 I was wondering if anyone can replicate this in their 
 environment or have an idea of how to determine the cause.

 An example is here:

 import std.c.stdio : printf;
 void main()
 {
     string variable = "variable";
     printf("made a string\n");

     string result;

     result = "before " ~ variable;
     printf("before\n");

     result = variable ~ " after";
     printf("after\n");

     result = "before " ~ "novar" ~ " after";
     printf("novar\n");

     result = "before " ~ variable ~ " after"; // this line will 
 seg fault
     printf("crashes before getting here\n");
 }
I check this. Please note that the Win64 version of LDC has still alpha quality. Every help is welcome here! Regards, Kai
Feb 24 2015
prev sibling next sibling parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Kevin Brogan"  wrote in message 
news:duzfvxzdoadpoxkxacuz forum.dlang.org...

 I've been able to successfully build LDC on 64 bit windows using msvc, but 
 I've noticed that a large number of test suites fail with a segmentation 
 fault instead of a regular error result.

 I've been able to determine that many of the segmentation faults is due to 
 the append operator.

 The fault only occurs when using a variable, and only when two append 
 operators are used in the same statement.

 I was wondering if anyone can replicate this in their environment or have 
 an idea of how to determine the cause.
The druntime function that implements the append operator uses variadic arguments (at least it does in DMD), so I would guess this is a symptom of them being broken on win64/LDC.
Feb 25 2015
parent "Kai Nacke" <kai redstar.de> writes:
On Wednesday, 25 February 2015 at 13:06:31 UTC, Daniel Murphy 
wrote:
 "Kevin Brogan"  wrote in message 
 news:duzfvxzdoadpoxkxacuz forum.dlang.org...

 I've been able to successfully build LDC on 64 bit windows 
 using msvc, but I've noticed that a large number of test 
 suites fail with a segmentation fault instead of a regular 
 error result.

 I've been able to determine that many of the segmentation 
 faults is due to the append operator.

 The fault only occurs when using a variable, and only when two 
 append operators are used in the same statement.

 I was wondering if anyone can replicate this in their 
 environment or have an idea of how to determine the cause.
The druntime function that implements the append operator uses variadic arguments (at least it does in DMD), so I would guess this is a symptom of them being broken on win64/LDC.
I merged the new vararg code yesterday. The example does not crash for me. Regards, Kai
Feb 25 2015
prev sibling next sibling parent reply "Johan Engelen" <j j.nl> writes:
On Wednesday, 25 February 2015 at 00:02:33 UTC, Kevin Brogan 
wrote:
 I've been able to successfully build LDC on 64 bit windows 
 using msvc, but I've noticed that a large number of test suites 
 fail with a segmentation fault instead of a regular error 
 result.
I'd like to help out with this. I also have LDC 64bit running on Windows using MSVC, but have not been able to run the tests. Or, ctest starts to work, but while building the (iirc) 3rd test no further output is given to the cmdline. There is an ldc2 process running on one core, and after an hour the ldc2 process has allocated more than 1.5 GB of memory... That's when I stopped it. Should I have more patience, or is there perhaps something wrong with the way I kick off the tests? I used http://wiki.dlang.org/Building_and_hacking_LDC_on_Windows_using_MS C#Running_the_tests and run ctest in the build folder (the ninja instructions end up not finding LDC2 in that folder (it does not search the path, but instead has a hardcoded location for LDC2.exe)).
Feb 25 2015
next sibling parent reply "Kevin Brogan" <kevin brogan.ca> writes:
On Wednesday, 25 February 2015 at 14:09:59 UTC, Johan Engelen 
wrote:

 I'd like to help out with this. I also have LDC 64bit running 
 on Windows using MSVC, but have not been able to run the tests.
I had to be careful about always running everything from within the visual studio native tools command prompt. I also had to put curl.lib in the environment lib path, (which was not defined at all on a fresh install), and put all of the LLVM and LDC bin directories in the path as well. Finally, I also had to modify the LLVM config file for both LDC and Ninja to pass /LARGEADDRESSAWARE:NO to the msvc linker or none of the debug builds would compile. c:\ldcenv\ninja-ldc2-x64\bin\ldc2.conf and %LDCROOT%\etc\ldc2.conf add ,"-L/LARGEADDRESSAWARE:NO" to switches Run ctest with -V as an option and it should give you some extra output. I'm not sure why it would just loop and consume memory though, but the third test does take a very long time to compile.
Feb 25 2015
parent reply "Kai Nacke" <kai redstar.de> writes:
Hi Kevin!

On Wednesday, 25 February 2015 at 19:09:44 UTC, Kevin Brogan 
wrote:
 Finally, I also had to modify the LLVM config file for both LDC 
 and Ninja to pass /LARGEADDRESSAWARE:NO to the msvc linker or 
 none of the debug builds would compile.

 c:\ldcenv\ninja-ldc2-x64\bin\ldc2.conf and
 %LDCROOT%\etc\ldc2.conf

 add    ,"-L/LARGEADDRESSAWARE:NO"    to switches
This is strange. /LARGEADDRESSAWARE:NO was only required for LLVM < 3.4. The Win64 version requires LLVM 3.6 otherwise there is no exception support. Regards, Kai
Feb 25 2015
next sibling parent "Kevin Brogan" <kevin brogan.ca> writes:
On Wednesday, 25 February 2015 at 20:47:14 UTC, Kai Nacke wrote:
 Hi Kevin!

 On Wednesday, 25 February 2015 at 19:09:44 UTC, Kevin Brogan 
 wrote:
 Finally, I also had to modify the LLVM config file for both 
 LDC and Ninja to pass /LARGEADDRESSAWARE:NO to the msvc linker 
 or none of the debug builds would compile.

 c:\ldcenv\ninja-ldc2-x64\bin\ldc2.conf and
 %LDCROOT%\etc\ldc2.conf

 add    ,"-L/LARGEADDRESSAWARE:NO"    to switches
This is strange. /LARGEADDRESSAWARE:NO was only required for LLVM < 3.4. The Win64 version requires LLVM 3.6 otherwise there is no exception support. Regards, Kai
Seems to have come back then. It's only required for debug builds. From what I can tell, this has something to do with the dwarf2 debug code using 32 bit relocations and msvc's link has some kind of bug in it. More info here: http://tortall.lighthouseapp.com/projects/78676/tickets/213-invalid-relocations-for-32-bit-dwarf2-in-64-bit-windows-object-files
Feb 25 2015
prev sibling parent "Johan Engelen" <j j.nl> writes:
On Wednesday, 25 February 2015 at 20:47:14 UTC, Kai Nacke wrote:
 Hi Kevin!

 On Wednesday, 25 February 2015 at 19:09:44 UTC, Kevin Brogan 
 wrote:
 Finally, I also had to modify the LLVM config file for both 
 LDC and Ninja to pass /LARGEADDRESSAWARE:NO to the msvc linker 
 or none of the debug builds would compile.

 c:\ldcenv\ninja-ldc2-x64\bin\ldc2.conf and
 %LDCROOT%\etc\ldc2.conf

 add    ,"-L/LARGEADDRESSAWARE:NO"    to switches
This is strange. /LARGEADDRESSAWARE:NO was only required for LLVM < 3.4. The Win64 version requires LLVM 3.6 otherwise there is no exception support.
I have the same /LARGEADDRESSAWARE:NO issue using LLVM 3.7.0svn-r230699.
Mar 01 2015
prev sibling parent "Kai Nacke" <kai redstar.de> writes:
Hi Johan!

On Wednesday, 25 February 2015 at 14:09:59 UTC, Johan Engelen 
wrote:
 On Wednesday, 25 February 2015 at 00:02:33 UTC, Kevin Brogan 
 wrote:
 I've been able to successfully build LDC on 64 bit windows 
 using msvc, but I've noticed that a large number of test 
 suites fail with a segmentation fault instead of a regular 
 error result.
I'd like to help out with this. I also have LDC 64bit running on Windows using MSVC, but have not been able to run the tests.
Finding a way how to reliable run the tests on Windows is an open issue. :-( Regards, Kai
Feb 25 2015
prev sibling parent reply "Kai Nacke" <kai redstar.de> writes:
On Wednesday, 25 February 2015 at 00:02:33 UTC, Kevin Brogan 
wrote:
 ...

 I've been able to determine that many of the segmentation 
 faults is due to the append operator.
Hi Kevin, are you already using the new vararg code? (Committed yesterday....) And which LLVM version do you use? I could not reproduce the crash on Win64 right now. Regards, Kai
Feb 25 2015
parent reply "Kevin Brogan" <kevin brogan.ca> writes:
On Wednesday, 25 February 2015 at 20:35:12 UTC, Kai Nacke wrote:
 On Wednesday, 25 February 2015 at 00:02:33 UTC, Kevin Brogan 
 wrote:
 ...

 I've been able to determine that many of the segmentation 
 faults is due to the append operator.
Hi Kevin, are you already using the new vararg code? (Committed yesterday....) And which LLVM version do you use? I could not reproduce the crash on Win64 right now. Regards, Kai
It's working for me now that I've pulled yesterday's vararg updates. Yay! Also, what timing!
Feb 25 2015
parent reply "kinke" <noone nowhere.com> writes:
 Yay!  Also, what timing!
Pure coincidence, but that append operator issue on Win64 was the initial reason for that PR. ;) What LLVM commit are you using? I'm using yesterday's release_36 head and exception handling on Win64 is sadly broken again, causing a myriad of failing tests. I can confirm the /LARGEADDRESSAWARE:NO issue. Wrt. running the tests: I often find myself taking a shortcut and running only the debug tests to reduce the required time considerably; phobos2-ldc-unittest takes a very long time, especially std.algorithm. You can disable the release tests by editing runtime\CTestTestfile.cmake in your build directory. Delete the first 6 non-comment lines and then all tests from core.atomic (new line 7, first druntime release test) to (but obviously excluding) core.atomic-debug (more or less at the center of the file). dmd-testsuite doesn't compile as it requires the make build system, so you don't need to edit tests\d2\CTestTestfile.cmake in a similar way. Beware that these ctest files get rebuilt as soon as your local LDC git head changes!
Feb 26 2015
parent reply "Kevin Brogan" <kevin brogan.ca> writes:
On Thursday, 26 February 2015 at 20:03:44 UTC, kinke wrote:
 Yay!  Also, what timing!
Pure coincidence, but that append operator issue on Win64 was the initial reason for that PR. ;) What LLVM commit are you using? I'm using yesterday's release_36 head and exception handling on Win64 is sadly broken again, causing a myriad of failing tests.
Exception handling broke on release_36 (or was always broken, not sure), so I went back to master. Currently exceptions are working for me on LLVM d89ac8f158976eaa9f9bcfba88f8a679771704f8 LDC 770b436e5c304ce635c186a09b04839f39fe4e71 I'm only failing 41 test cases and I've just about to post about how to fix an assembly optimization bug that will bring that number down to 40 soon i hope. With LLVM I've also got a custom change to Support/raw_ostream.cpp and Support/ErrorHandling.cpp of # include <unistd.h> # if !defined(HAVE_STDINT_H) # include <unistd.h> # endif To remove a compilation error about redefinitions.
 I can confirm the /LARGEADDRESSAWARE:NO issue.

 Wrt. running the tests: I often find myself taking a shortcut 
 and running only the debug tests to reduce the required time 
 considerably; phobos2-ldc-unittest takes a very long time, 
 especially std.algorithm. You can disable the release tests by 
 editing runtime\CTestTestfile.cmake in your build directory. 
 Delete the first 6 non-comment lines and then all tests from 
 core.atomic (new line 7, first druntime release test) to (but 
 obviously excluding) core.atomic-debug (more or less at the 
 center of the file).
 dmd-testsuite doesn't compile as it requires the make build 
 system, so you don't need to edit tests\d2\CTestTestfile.cmake 
 in a similar way.
 Beware that these ctest files get rebuilt as soon as your local 
 LDC git head changes!
Bloody thing does take forever. Little over an hour on my machine. I'm working on test 19 at the moment so I use ctest's options to run just what is needed. If I want to modify command line options for test 1 or 2, I copy and paste the output from ctest --force-new-ctest-process --output-on-failure -I 1,2,1,19 -V (added verbose flag) into a batch file. Test two is too long for the command prompt (maximum input size... wtf). I modify the runtime command by adding "-L/LIBPATH:C:/ldcenv/ninja-ldc2-x64/runtime/src/" and then removing the base path i just added from all of the object files. ctest --force-new-ctest-process --output-on-failure -I 1,2,1,19
Feb 26 2015
next sibling parent "kinke" <noone nowhere.com> writes:
 LLVM d89ac8f158976eaa9f9bcfba88f8a679771704f8
Thank you very much, this one works for me too with current LDC head. Compiling the tests right now...
 I'm only failing 41 test cases and I've just about to post 
 about how to fix an assembly optimization bug that will bring 
 that number down to 40 soon i hope.
Perfect, looking forward to it. :) Note that there's https://github.com/ldc-developers/ldc/issues/758 to keep track of these failing tests on Win64. E.g., we've also had 41 failures on October 25th 2014.
Feb 27 2015
prev sibling parent "Johan Engelen" <j j.nl> writes:
Thanks for your hints. I am now able to run (part of) the tests, 
and have enough leads to make them work and start working on 
getting some tests to pass :)

A very happy,
   Johan
Feb 27 2015