www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - D-thrift package detects regressions since 2.061, where is the

reply "glycerine" <noreply noreply.com> writes:
Grrr...

Apparently nobody has been testing the D - Apache Thrift bindings
since 2.061, and dmd has since accumulated multiple regressions
that affect the correctness of the Thrift implementation. I
emailed with David N. and he said that this was quite common for
each release of dmd, and that while he used to religously
evaluate each new dmd release on the Thrift bindings, he had
simply not had the time for more recent recents to test each
thoroughly.

Serialization: this is fundamental. This really isn't the kind of
thing that should ever be allowed to break. Hence it really isn't
something that should be tested manually. It should be an
automatic part of the automatic regression detection test suite.

Where is the regression suite for D located, and how can I add to
it?

There used to be github issue tracking, but I don't see it any
more... is it hiding under their new interface perhaps...?

Thanks.

- glycerine
Aug 13 2013
next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Tuesday, August 13, 2013 19:49:51 glycerine wrote:
 Grrr...
 
 Apparently nobody has been testing the D - Apache Thrift bindings
 since 2.061, and dmd has since accumulated multiple regressions
 that affect the correctness of the Thrift implementation. I
 emailed with David N. and he said that this was quite common for
 each release of dmd, and that while he used to religously
 evaluate each new dmd release on the Thrift bindings, he had
 simply not had the time for more recent recents to test each
 thoroughly.
 
 Serialization: this is fundamental. This really isn't the kind of
 thing that should ever be allowed to break. Hence it really isn't
 something that should be tested manually. It should be an
 automatic part of the automatic regression detection test suite.
 
 Where is the regression suite for D located, and how can I add to
 it?

We do not include 3rd party libraries or projects in any kind of regression suite, so if that's what you're looking for, you're out of luck. David or someone else working on the Thrift stuff would have had to set something up for the D Thrift stuff specifically. We do have an autotester which tests that the current compiler and standard library implementation pass their tests, as well as a tester which tests that pull requests pass the tests as well, and that can be found here: http://d.puremagic.com/test-results/ If you report bugs in bugzilla, then when they are fixed, unit tests for those bugs will be added for them so that they won't fail again. Our bugzilla can be found here: http://d.puremagic.com/issues.
 There used to be github issue tracking, but I don't see it any
 more... is it hiding under their new interface perhaps...?

We've never used github issue tracking for either the compiler or D's standard libraries. Maybe they were enabled before, but if so, they were ignored. I don't know what the D Thrift stuff does though. - Jonathan M Davis
Aug 13 2013
prev sibling next sibling parent reply Denis Shelomovskij <verylonglogin.reg gmail.com> writes:
13.08.2013 21:49, glycerine пишет:
 Grrr...

 Apparently nobody has been testing the D - Apache Thrift bindings
 since 2.061, and dmd has since accumulated multiple regressions
 that affect the correctness of the Thrift implementation. I
 emailed with David N. and he said that this was quite common for
 each release of dmd, and that while he used to religously
 evaluate each new dmd release on the Thrift bindings, he had
 simply not had the time for more recent recents to test each
 thoroughly.

 Serialization: this is fundamental. This really isn't the kind of
 thing that should ever be allowed to break. Hence it really isn't
 something that should be tested manually. It should be an
 automatic part of the automatic regression detection test suite.

 Where is the regression suite for D located, and how can I add to
 it?

 There used to be github issue tracking, but I don't see it any
 more... is it hiding under their new interface perhaps...?

 Thanks.

 - glycerine

By the way, the ability to add costume projects to D autotester is already proposed without any response: http://forum.dlang.org/thread/kqm4ta$m7f$1 digitalmars.com -- Денис В. Шеломовский Denis V. Shelomovskij
Aug 23 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/23/2013 10:34 AM, Denis Shelomovskij wrote:
 By the way, the ability to add costume projects to D autotester is already
 proposed without any response:
 http://forum.dlang.org/thread/kqm4ta$m7f$1 digitalmars.com

The question comes up repeatedly, and I've answered it repeatedly, the latest on 8/20 in the thread "std.serialization: pre-voting review / discussion". Here's the message: --------------------------------- On 8/18/2013 9:33 AM, David Nadlinger wrote:
 Having a system that regularly, automatically runs the test suites of several
 larger, well-known D projects with the results being readily available to the
 DMD/druntime/Phobos teams would certainly help. But it's also not ideal, since
 if a project starts to fail, the exact nature of the issue (regression in DMD
or
 bug in the project, and if the former, a minimal test case) can often be hard
to
 track down for somebody not already familiar with the code base.

That's exactly the problem. If these large projects are incorporated into the autotester, who is going to isolate/fix problems arising with them? The test suite is designed to be a collection of already-isolated issues, so understanding what went wrong shouldn't be too difficult. Note that already it is noticeably much harder to debug a phobos unit test gone awry than the other tests. A full blown project that nobody understands would fare far worse. (And the other problem, of course, is the test suite is designed to be runnable fairly quickly. Compiling some other large project and running its test suite can make the autotester much less useful when the turnaround time increases.) Putting large projects into the autotester has the implication that development and support of those projects has been ceded to the core dev team, i.e. who is responsible for it has been badly blurred.
Aug 23 2013
parent reply David <d dav1d.de> writes:
Am 23.08.2013 20:07, schrieb Walter Bright:
 On 8/23/2013 10:34 AM, Denis Shelomovskij wrote:
 By the way, the ability to add costume projects to D autotester is
 already
 proposed without any response:
 http://forum.dlang.org/thread/kqm4ta$m7f$1 digitalmars.com

The question comes up repeatedly, and I've answered it repeatedly, the latest on 8/20 in the thread "std.serialization: pre-voting review / discussion". Here's the message: --------------------------------- On 8/18/2013 9:33 AM, David Nadlinger wrote:
 Having a system that regularly, automatically runs the test suites of

 larger, well-known D projects with the results being readily available

 DMD/druntime/Phobos teams would certainly help. But it's also not

 if a project starts to fail, the exact nature of the issue (regression

 bug in the project, and if the former, a minimal test case) can often

 track down for somebody not already familiar with the code base.

That's exactly the problem. If these large projects are incorporated into the autotester, who is going to isolate/fix problems arising with them? The test suite is designed to be a collection of already-isolated issues, so understanding what went wrong shouldn't be too difficult. Note that already it is noticeably much harder to debug a phobos unit test gone awry than the other tests. A full blown project that nobody understands would fare far worse. (And the other problem, of course, is the test suite is designed to be runnable fairly quickly. Compiling some other large project and running its test suite can make the autotester much less useful when the turnaround time increases.) Putting large projects into the autotester has the implication that development and support of those projects has been ceded to the core dev team, i.e. who is responsible for it has been badly blurred.

I find it funny how hard you try to get D "production ready" and make (in my opinion) bad decisions affecting the future of D, but I hit every release at least 3 regressions, 1 of these is mostly a real WTF. At least run this test-suite consisting of 3rd party projects every now and then and especially before a release. I personally don't mind these regressions (well I do, but I can live with them), trying to find the root for a day or two and fix it there, but companies will. While we're at it, the D release-zip should also be checked, not beeing able to install a new release because the zip is fucked up doesn't speak for D (especially when it takes forever to download dmd for windows and you find a shitton of binaries for other OSes as well) </rant>
Aug 23 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/23/2013 3:11 PM, David wrote:
 I find it funny how hard you try to get D "production ready" and make
 (in my opinion) bad decisions affecting the future of D, but I hit every
 release at least 3 regressions, 1 of these is mostly a real WTF.

Please join us in the beta test program, then. The point of it is so that users can compile their projects and find problems before we do a release.
Aug 23 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Aug 23, 2013 at 11:07:35AM -0700, Walter Bright wrote:
 On 8/23/2013 10:34 AM, Denis Shelomovskij wrote:
By the way, the ability to add costume projects to D autotester is
already proposed without any response:
http://forum.dlang.org/thread/kqm4ta$m7f$1 digitalmars.com

The question comes up repeatedly, and I've answered it repeatedly, the latest on 8/20 in the thread "std.serialization: pre-voting review / discussion". Here's the message: --------------------------------- On 8/18/2013 9:33 AM, David Nadlinger wrote:
 Having a system that regularly, automatically runs the test suites
 of several larger, well-known D projects with the results being
 readily available to the DMD/druntime/Phobos teams would certainly
 help. But it's also not ideal, since if a project starts to fail,
 the exact nature of the issue (regression in DMD or bug in the
 project, and if the former, a minimal test case) can often be hard
 to track down for somebody not already familiar with the code base.

That's exactly the problem. If these large projects are incorporated into the autotester, who is going to isolate/fix problems arising with them? The test suite is designed to be a collection of already-isolated issues, so understanding what went wrong shouldn't be too difficult. Note that already it is noticeably much harder to debug a phobos unit test gone awry than the other tests. A full blown project that nobody understands would fare far worse. (And the other problem, of course, is the test suite is designed to be runnable fairly quickly. Compiling some other large project and running its test suite can make the autotester much less useful when the turnaround time increases.) Putting large projects into the autotester has the implication that development and support of those projects has been ceded to the core dev team, i.e. who is responsible for it has been badly blurred.

One idea that occurred to me is to put large external projects under a separate tester, not bound to the core dmd/druntime/phobos autotesting, but an independent tester that regularly checks out git HEAD and compiles & tests said large projects. The devs can then monitor the status of these tests independently, and when something goes wrong, they can ping whoever is responsible for that project to investigate what might be the cause. If it's caused by the latest git commit(s), they can file regression bugs on the bugtracker, otherwise, they update their code to work with the new compiler. If the responsible person doesn't respond, or there is no contact person, we could use this forum as a kind of community notice that something needs to be fixed somewhere. If nobody responds, the project is likely not worth keeping up with. This way we don't slow down development / autotesting unnecessarily, and still let the community know when there might be potential problems with existing code. (It probably also helps for the core devs to be aware of potential regressions without being held back from code changes.) If it's important, *somebody* will step up and file bugs and/or fix the issue. If nobody cares, then probably it's not worth fretting over. T -- Do not reason with the unreasonable; you lose by definition.
Aug 23 2013
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 8/23/13, H. S. Teoh <hsteoh quickfur.ath.cx> wrote:
 The devs can then monitor the
 status of these tests independently, and when something goes wrong, they
 can ping whoever is responsible for that project to investigate what
 might be the cause.

That's basically the same thing I said in that other thread. Considering that we have a very close D community, this sort of workflow could work.
Aug 23 2013
prev sibling next sibling parent "David Nadlinger" <code klickverbot.at> writes:
On Friday, 23 August 2013 at 20:13:21 UTC, Andrej Mitrovic wrote:
 On 8/23/13, H. S. Teoh <hsteoh quickfur.ath.cx> wrote:
 The devs can then monitor the
 status of these tests independently, and when something goes 
 wrong, they
 can ping whoever is responsible for that project to 
 investigate what
 might be the cause.

That's basically the same thing I said in that other thread. Considering that we have a very close D community, this sort of workflow could work.

Yep. I don't think anybody is suggesting to add external projects to the main CI system – at least I certainly wouldn't, being part of both the compiler and library writer camps. But it could still be useful to have an automated "real world health check" that is available publicly, maybe even integrated with the official D package repository, if we are going to have one anytime soon (btw, did I miss any "official" announcement regarding code.dlang.org/dub?). This is not about shifting the responsibility for maintaining the library to the core team, it's about empowering people working on the frontend/druntime/Phobos to know when they are breaking real-world code. I think it's pretty clear at this point that the regression test suite is not anywhere near exhaustive for even the specified parts of the language, and then there are still the many holes in the spec to consider as well. Maybe we can pull this off with an improved beta process where all D users are actually persuaded to participate. But seeing that Walter seems to avoid incorporating any of the ideas in that direction that have been brought up on dmd-beta previously (more public announcements, at least a rough tentative release schedule, meaningful file names/version info, …), I'm not sure what the plan is there. David
Aug 23 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Aug 24, 2013 at 01:18:14AM +0200, David Nadlinger wrote:
[...]
 Maybe we can pull this off with an improved beta process where all D
 users are actually persuaded to participate. But seeing that Walter
 seems to avoid incorporating any of the ideas in that direction that
 have been brought up on dmd-beta previously (more public
 announcements, at least a rough tentative release schedule, meaningful
 file names/version info, …), I'm not sure what the plan is there.

It could be as simple as announcing the availability of a beta release on the main D forum / newsgroup (i.e. here). I didn't even know dmd-beta existed until more than a year after I joined the D community. And now that I'm subscribed, I hardly ever hear anything from it. Posting the beta announcement here instead would, at the very least, reach a far wider audience. (Or maybe post in both places. Wouldn't hurt. The whole point of a beta *release* is for the world to know about it so that it can try things out before the actual release. No point hiding the announcement in some obscure isolated corner.) T -- Spaghetti code may be tangly, but lasagna code is just cheesy.
Aug 23 2013
prev sibling next sibling parent Val Markovic <val markovic.io> writes:
--001a113368763169f104e4a60d18
Content-Type: text/plain; charset=UTF-8

On Fri, Aug 23, 2013 at 11:07 AM, Walter Bright
<newshound2 digitalmars.com>wrote:

 That's exactly the problem. If these large projects are incorporated into
 the autotester, who is going to isolate/fix problems arising with them?

 The test suite is designed to be a collection of already-isolated issues,
 so understanding what went wrong shouldn't be too difficult. Note that
 already it is noticeably much harder to debug a phobos unit test gone awry
 than the other tests. A full blown project that nobody understands would
 fare far worse.

AFAIR both Clang and GCC have entire third-party projects in their test suite. I know that at least SQLite is part of both, and that's a pretty big project. If I recall correctly, GCC releases are blocked on successfully compiling the Linux kernel, all of Firefox and I possibly Qt. The third-party project tests need to finish without failures as well. My recollection is a bit vague here though. Now, do they compile and run all the tests for these projects on every commit or make sure nothing has broken just before making a new release, I don't know. But I do know that it's the latter at the very least. --001a113368763169f104e4a60d18 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">On Fri, Aug 23, 2013 at 11:07 AM, Walter Bright <span dir= =3D"ltr">&lt;<a href=3D"mailto:newshound2 digitalmars.com" target=3D"_blank= ">newshound2 digitalmars.com</a>&gt;</span> wrote:<br><div class=3D"gmail_e= xtra"><div class=3D"gmail_quote"> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> That&#39;s exactly the problem. If these large projects are incorporated in= to the autotester, who is going to isolate/fix problems arising with them?<= br> <br> The test suite is designed to be a collection of already-isolated issues, s= o understanding what went wrong shouldn&#39;t be too difficult. Note that a= lready it is noticeably much harder to debug a phobos unit test gone awry t= han the other tests. A full blown project that nobody understands would far= e far worse.<br> </blockquote><div><br></div><div>AFAIR both Clang and GCC have entire third= -party projects in their test suite. I know that at least SQLite is part of= both, and that&#39;s a pretty big project. If I recall correctly, GCC rele= ases are blocked on successfully compiling the Linux kernel, all of Firefox= and I possibly Qt. The third-party project tests need to finish without fa= ilures as well.=C2=A0</div> <div><br></div><div>My recollection is a bit vague here though.</div><div><= br></div><div>Now, do they compile and run all the tests for these projects= on every commit or make sure nothing has broken just before making a new r= elease, I don&#39;t know. But I do know that it&#39;s the latter at the ver= y least.</div> </div></div></div> --001a113368763169f104e4a60d18--
Aug 23 2013
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 23/08/13 20:38, H. S. Teoh wrote:
 One idea that occurred to me is to put large external projects under a
 separate tester, not bound to the core dmd/druntime/phobos autotesting,
 but an independent tester that regularly checks out git HEAD and
 compiles & tests said large projects.

I proposed something along these lines shortly after DConf: http://forum.dlang.org/thread/mailman.47.1369319426.13711.digitalmars-d puremagic.com I thought it could be useful both as a stability tester _and_ as a means to evaluate the prospective impact of deliberately breaking changes. It was quite well received as an idea but is probably a big job to take on ... :-(
Aug 24 2013
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Saturday, 24 August 2013 at 07:50:00 UTC, Joseph Rushton 
Wakeling wrote:
 I proposed something along these lines shortly after DConf:
 http://forum.dlang.org/thread/mailman.47.1369319426.13711.digitalmars-d puremagic.com

 I thought it could be useful both as a stability tester _and_ 
 as a means to evaluate the prospective impact of deliberately 
 breaking changes.

 It was quite well received as an idea but is probably a big job 
 to take on ... :-(

I do want to contribute one once I decide what I want to do about a more powerful server (such suite is a bit too hard for my small VPS). Pretty sure something can be done this year, but will take some time (months+).
Aug 25 2013
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 25/08/13 21:57, Dicebot wrote:
 I do want to contribute one once I decide what I want to do about a more
 powerful server (such suite is a bit too hard for my small VPS). Pretty sure
 something can be done this year, but will take some time (months+).

Is this something where we can do some crowdfunding? Might be worth trying to invest in some server/cloud infrastructure for D projects like this that have collective benefit.
Aug 25 2013
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Sunday, 25 August 2013 at 20:02:49 UTC, Joseph Rushton 
Wakeling wrote:
 On 25/08/13 21:57, Dicebot wrote:
 I do want to contribute one once I decide what I want to do 
 about a more
 powerful server (such suite is a bit too hard for my small 
 VPS). Pretty sure
 something can be done this year, but will take some time 
 (months+).

Is this something where we can do some crowdfunding? Might be worth trying to invest in some server/cloud infrastructure for D projects like this that have collective benefit.

Well, I wanted to do the move anyway for my own needs so it is just a planned side effect. It is not that costly, I simply don't have time to do the move properly.
Aug 25 2013