digitalmars.D - Adding more projects to the Project Tester
- Seb (29/29) Jul 05 2018 So learning from the recent Vibe.d regression fiasco (we
- Joakim (9/38) Jul 05 2018 The LDC compiler? kinke recently had an issue because of all the
- Nicholas Wilson (3/11) Jul 06 2018 Thats a great idea! Especially since there is a whole lot more
- Seb (3/11) Jul 08 2018 Good idea. Not sure how easy/hard it is, but let's give it a try:
- tide (2/31) Jul 06 2018 Include Windows as part of the testing done.
- Seb (19/59) Jul 08 2018 Just to avoid confusion, testing of DMD/Druntime/Phobos/etc. is
- JN (7/12) Jul 06 2018 By the way, is there any policy for outdated dub packages? There
- Basile B. (2/11) Jul 06 2018 You just found an idea for the score algorithm.
- Guillaume Piolat (4/16) Jul 07 2018 Please include the intel-intrinsics package, its test expose a
- Seb (2/5) Jul 07 2018 https://github.com/dlang/ci/pull/224
- Guillaume Piolat (5/9) Jul 07 2018 A meta-problem is that many package unittest (at least on typical
- Jonathan M Davis (8/37) Jul 08 2018 dxml should probably be on the list. Assuming that this is only on *nix ...
- Seb (9/20) Jul 09 2018 Yes!
- Jonathan M Davis (4/14) Jul 09 2018 Ah. Then I will have to figure out what needs to go in .travis.yml and a...
So learning from the recent Vibe.d regression fiasco (we temporarily disabled a subconfiguration in Vibe.d and promptly got a regression in 2.081), I think we should try to add more projects to the Project Tester. The current list is here: https://github.com/dlang/ci/blob/master/vars/runPipeline.groovy#L443 Any suggestions? Why should I add my project to the Project Tester? -------------------------------------------------- Once a project is added to the Project Tester, DMD can't regress on it anymore as for every PR at dmd, druntime, phobos, tools and dub the testsuite of the added projects are run. How does the Project Tester work? --------------------------------- - By default, it will run the same commands as Travis would do. Although, if necessary, custom commands can be used too. - It will checkout the latest, stable git tag Requirements ------------ - moderately popular or was prone to regressions in the past - rather easy to build (i.e. you don't need to download and recompile clang) - no flaky testsuite (random errors in the testsuite due to network connectivity shouldn't happen. Though there's `DETERMINISTIC_HINT=1` set, s.t. you could disable such parts of your testsuite) - reachable author or development (if there's ever a case where we need to push changes via a trivial PR to the repo, it shouldn't sit in the queue for weeks)
Jul 05 2018
On Friday, 6 July 2018 at 03:19:44 UTC, Seb wrote:So learning from the recent Vibe.d regression fiasco (we temporarily disabled a subconfiguration in Vibe.d and promptly got a regression in 2.081), I think we should try to add more projects to the Project Tester. The current list is here: https://github.com/dlang/ci/blob/master/vars/runPipeline.groovy#L443 Any suggestions? Why should I add my project to the Project Tester? -------------------------------------------------- Once a project is added to the Project Tester, DMD can't regress on it anymore as for every PR at dmd, druntime, phobos, tools and dub the testsuite of the added projects are run. How does the Project Tester work? --------------------------------- - By default, it will run the same commands as Travis would do. Although, if necessary, custom commands can be used too. - It will checkout the latest, stable git tag Requirements ------------ - moderately popular or was prone to regressions in the past - rather easy to build (i.e. you don't need to download and recompile clang) - no flaky testsuite (random errors in the testsuite due to network connectivity shouldn't happen. Though there's `DETERMINISTIC_HINT=1` set, s.t. you could disable such parts of your testsuite) - reachable author or development (if there's ever a case where we need to push changes via a trivial PR to the repo, it shouldn't sit in the queue for weeks)The LDC compiler? kinke recently had an issue because of all the C++ integration changes upstream: https://github.com/ldc-developers/ldc/pull/2752#issuecomment-398897813 As perhaps the largest consumer of extern(C++), it may make sense to add it for all the C++ work being done. It would require the llvm package be pre-installed in the test environ. List of current projects looks great, was tough to think of anything to add.
Jul 05 2018
Nicholas Wilson <iamthewilsonator hotmail.com> On Friday, 6 July 2018 at 05:02:56 UTC, Joakim wrote:The LDC compiler? kinke recently had an issue because of all the C++ integration changes upstream: https://github.com/ldc-developers/ldc/pull/2752#issuecomment-398897813 As perhaps the largest consumer of extern(C++), it may make sense to add it for all the C++ work being done. It would require the llvm package be pre-installed in the test environ. List of current projects looks great, was tough to think of anything to add.Thats a great idea! Especially since there is a whole lot more extern(C++) stuff coming in the next DMD release.
Jul 06 2018
On Friday, 6 July 2018 at 05:02:56 UTC, Joakim wrote:The LDC compiler? kinke recently had an issue because of all the C++ integration changes upstream: https://github.com/ldc-developers/ldc/pull/2752#issuecomment-398897813 As perhaps the largest consumer of extern(C++), it may make sense to add it for all the C++ work being done. It would require the llvm package be pre-installed in the test environ. List of current projects looks great, was tough to think of anything to add.Good idea. Not sure how easy/hard it is, but let's give it a try: https://github.com/dlang/ci/pull/228
Jul 08 2018
On Friday, 6 July 2018 at 03:19:44 UTC, Seb wrote:So learning from the recent Vibe.d regression fiasco (we temporarily disabled a subconfiguration in Vibe.d and promptly got a regression in 2.081), I think we should try to add more projects to the Project Tester. The current list is here: https://github.com/dlang/ci/blob/master/vars/runPipeline.groovy#L443 Any suggestions? Why should I add my project to the Project Tester? -------------------------------------------------- Once a project is added to the Project Tester, DMD can't regress on it anymore as for every PR at dmd, druntime, phobos, tools and dub the testsuite of the added projects are run. How does the Project Tester work? --------------------------------- - By default, it will run the same commands as Travis would do. Although, if necessary, custom commands can be used too. - It will checkout the latest, stable git tag Requirements ------------ - moderately popular or was prone to regressions in the past - rather easy to build (i.e. you don't need to download and recompile clang) - no flaky testsuite (random errors in the testsuite due to network connectivity shouldn't happen. Though there's `DETERMINISTIC_HINT=1` set, s.t. you could disable such parts of your testsuite) - reachable author or development (if there's ever a case where we need to push changes via a trivial PR to the repo, it shouldn't sit in the queue for weeks)Include Windows as part of the testing done.
Jul 06 2018
On Friday, 6 July 2018 at 21:25:28 UTC, tide wrote:On Friday, 6 July 2018 at 03:19:44 UTC, Seb wrote:Just to avoid confusion, testing of DMD/Druntime/Phobos/etc. is done on Windows: https://auto-tester.puremagic.com/platform-history.ghtml?projectid=1&os=Win_32 https://auto-tester.puremagic.com/platform-history.ghtml?projectid=1&os=Win_32_64 https://ci.appveyor.com/project/greenify/dmd ... In my experience are the huge majority of all regressions originating in the D frontend where architecture actually doesn't matter so much. That being said, we are currently investigating to move the Project Tester from Jenkins to Buildkite (https://github.com/dlang/ci/pull/225) and one advantage of this move is that it'll be a lot easier to add new machines to the build cloud (so you could then hook up that Windows machine in your garage with the Project Tester). However, AFAICT many of the dub projects do have dependencies that are harder to install on Windows, so this might not happen in the near future.So learning from the recent Vibe.d regression fiasco (we temporarily disabled a subconfiguration in Vibe.d and promptly got a regression in 2.081), I think we should try to add more projects to the Project Tester. The current list is here: https://github.com/dlang/ci/blob/master/vars/runPipeline.groovy#L443 Any suggestions? Why should I add my project to the Project Tester? -------------------------------------------------- Once a project is added to the Project Tester, DMD can't regress on it anymore as for every PR at dmd, druntime, phobos, tools and dub the testsuite of the added projects are run. How does the Project Tester work? --------------------------------- - By default, it will run the same commands as Travis would do. Although, if necessary, custom commands can be used too. - It will checkout the latest, stable git tag Requirements ------------ - moderately popular or was prone to regressions in the past - rather easy to build (i.e. you don't need to download and recompile clang) - no flaky testsuite (random errors in the testsuite due to network connectivity shouldn't happen. Though there's `DETERMINISTIC_HINT=1` set, s.t. you could disable such parts of your testsuite) - reachable author or development (if there's ever a case where we need to push changes via a trivial PR to the repo, it shouldn't sit in the queue for weeks)Include Windows as part of the testing done.
Jul 08 2018
On Friday, 6 July 2018 at 03:19:44 UTC, Seb wrote:Why should I add my project to the Project Tester? -------------------------------------------------- Once a project is added to the Project Tester, DMD can't regress on it anymore as for every PR at dmd, druntime, phobos, tools and dub the testsuite of the added projects are run.By the way, is there any policy for outdated dub packages? There are some packages in dub registry, which don't build with the latest D version anymore. Should such packages stay in the registry? It gives a bit inflated view of packages, seems like there are many, but 50% (for example) aren't even usable without additional effort.
Jul 06 2018
On Friday, 6 July 2018 at 21:47:34 UTC, JN wrote:On Friday, 6 July 2018 at 03:19:44 UTC, Seb wrote:You just found an idea for the score algorithm.Why should I add my project to the Project Tester? -------------------------------------------------- Once a project is added to the Project Tester, DMD can't regress on it anymore as for every PR at dmd, druntime, phobos, tools and dub the testsuite of the added projects are run.By the way, is there any policy for outdated dub packages?
Jul 06 2018
On Friday, 6 July 2018 at 23:56:01 UTC, Basile B. wrote:On Friday, 6 July 2018 at 21:47:34 UTC, JN wrote:Why isn't there something like "compiler compatibility" in a dub config file? E.g. currently the vibe.d project lists the compiler versions the code is compatible with [1]. Wouldn't a field in dub like "compilerCompatibility": { "dmd": "2.080.0", "ldc": "..." } be nice? This could also help code.dlang.org to show compatibility of libraries/packages. The only pain I guess would be to keep such a list up to date. [1]: https://github.com/vibe-d/vibe.d#supportBy the way, is there any policy for outdated dub packages?You just found an idea for the score algorithm.
Jul 07 2018
On Saturday, 7 July 2018 at 08:04:47 UTC, Timoses wrote:On Friday, 6 July 2018 at 23:56:01 UTC, Basile B. wrote:I'm not sure it's a good idea to manually hard-code such a list into the dub.json. However, for every release or so, the dub registry could crawl all its packages and run `dub test` on them. Of course, there's the chance that a build still fails (e.g. due to missing dependencies) even though it would pass locally, but each package that passes with the latest registry could receive additional points.On Friday, 6 July 2018 at 21:47:34 UTC, JN wrote:Why isn't there something like "compiler compatibility" in a dub config file? E.g. currently the vibe.d project lists the compiler versions the code is compatible with [1]. Wouldn't a field in dub like "compilerCompatibility": { "dmd": "2.080.0", "ldc": "..." } be nice? This could also help code.dlang.org to show compatibility of libraries/packages. The only pain I guess would be to keep such a list up to date. [1]: https://github.com/vibe-d/vibe.d#supportBy the way, is there any policy for outdated dub packages?You just found an idea for the score algorithm.
Jul 08 2018
On Friday, 6 July 2018 at 03:19:44 UTC, Seb wrote:Requirements ------------ - moderately popular or was prone to regressions in the past - rather easy to build (i.e. you don't need to download and recompile clang) - no flaky testsuite (random errors in the testsuite due to network connectivity shouldn't happen. Though there's `DETERMINISTIC_HINT=1` set, s.t. you could disable such parts of your testsuite) - reachable author or development (if there's ever a case where we need to push changes via a trivial PR to the repo, it shouldn't sit in the queue for weeks)Please include the intel-intrinsics package, its test expose a regression on 2.081 that I haven't looked after yet. The goal is to provide a _stable and usable_ SIMD interface.
Jul 07 2018
On Saturday, 7 July 2018 at 16:29:15 UTC, Guillaume Piolat wrote:Please include the intel-intrinsics package, its test expose a regression on 2.081 that I haven't looked after yet. The goal is to provide a _stable and usable_ SIMD interface.https://github.com/dlang/ci/pull/224
Jul 07 2018
On Friday, 6 July 2018 at 03:19:44 UTC, Seb wrote:So learning from the recent Vibe.d regression fiasco (we temporarily disabled a subconfiguration in Vibe.d and promptly got a regression in 2.081), I think we should try to add more projects to the Project Tester.A meta-problem is that many package unittest (at least on typical CI) don't exercise the DMD optimizer since they don't run with -O or -release. Dub has no default options to dub test with optimizations so ti's very rarely tested.
Jul 07 2018
On Thursday, 5 July 2018 21:19:44 MDT Seb via Digitalmars-d wrote:So learning from the recent Vibe.d regression fiasco (we temporarily disabled a subconfiguration in Vibe.d and promptly got a regression in 2.081), I think we should try to add more projects to the Project Tester. The current list is here: https://github.com/dlang/ci/blob/master/vars/runPipeline.groovy#L443 Any suggestions? Why should I add my project to the Project Tester? -------------------------------------------------- Once a project is added to the Project Tester, DMD can't regress on it anymore as for every PR at dmd, druntime, phobos, tools and dub the testsuite of the added projects are run. How does the Project Tester work? --------------------------------- - By default, it will run the same commands as Travis would do. Although, if necessary, custom commands can be used too. - It will checkout the latest, stable git tag Requirements ------------ - moderately popular or was prone to regressions in the past - rather easy to build (i.e. you don't need to download and recompile clang) - no flaky testsuite (random errors in the testsuite due to network connectivity shouldn't happen. Though there's `DETERMINISTIC_HINT=1` set, s.t. you could disable such parts of your testsuite) - reachable author or development (if there's ever a case where we need to push changes via a trivial PR to the repo, it shouldn't sit in the queue for weeks)dxml should probably be on the list. Assuming that this is only on *nix and not Windows (I have yet to write a test script for Windows), then ./test.sh will run the tests without -release, and ./testRelease.sh will run the tests with -release (simply running dub test won't do anything useful, because all of the unittest blocks are versioned to avoid problems when projects depending on dxml compile their tests). - Jonathan M Davis
Jul 08 2018
On Monday, 9 July 2018 at 05:58:17 UTC, Jonathan M Davis wrote:On Thursday, 5 July 2018 21:19:44 MDT Seb via Digitalmars-d wrote:Yes! https://github.com/dlang/ci/pull/230[...]dxml should probably be on the list.Assuming that this is only on *nix and not WindowsYes.(I have yet to write a test script for Windows)This might help: https://github.com/Abscissa/AppVeyor-Dthen ./test.sh will run the tests without -release, and ./testRelease.sh will run the tests with -release (simply running dub test won't do anything useful, because all of the unittest blocks are versioned to avoid problems when projects depending on dxml compile their tests).The Project Tester will by default look at (if existent) the .travis.yml and run the same 'script' commands, so it should do the right thing, but I will keep this in mind. Thanks for the explanation!
Jul 09 2018
On Monday, 9 July 2018 02:44:31 MDT Seb via Digitalmars-d wrote:On Monday, 9 July 2018 at 05:58:17 UTC, Jonathan M Davis wrote:Ah. Then I will have to figure out what needs to go in .travis.yml and add it. - Jonathan M Davisthen ./test.sh will run the tests without -release, and ./testRelease.sh will run the tests with -release (simply running dub test won't do anything useful, because all of the unittest blocks are versioned to avoid problems when projects depending on dxml compile their tests).The Project Tester will by default look at (if existent) the .travis.yml and run the same 'script' commands, so it should do the right thing, but I will keep this in mind. Thanks for the explanation!
Jul 09 2018