www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Time for Phobos CTFE-ability unittests...right? RIGHT?

reply "Nick Sabalausky" <a a.a> writes:
Yet again, a new DMD release has broken my code, and other people's code, 
too, just because of Phobos functions loosing their CTFE-ability. (strip(), 
toLower(), etc... And yes, I did bring up the strip() regression on the beta 
list, to no effect.)

We praise and promote the hell out of CTFE (and rightly so) for being able 
to take ordinary functions and run them at compile-time. Except, with every 
new release, I become more and more convinced that Phobos should *not* be 
used in CTFE, because I know from experience that if I do, chances are it 
will soon break. So we have to maintain separate CTFE-guaranteed versions of 
whatever we need from Phobos. Obviously, aside from being a royal pain in 
the ass,  that completely defeats the point of CTFE. Worse still, it makes 
us look really, really bad when we go out and promote a CTFE that just plain 
doesn't work as advertised when it comes the language's own standard 
library.

Granted, I *completely* understand and sympathize with the practical issues 
involved (Phobos being heavily in-development, CTFE itself undergoing big 
improvements, etc...) So I fully agree it made perfect sense for this 
"bleeding-edge/unstable" branch of D to not concern itself with CTFE-ability 
regressions in Phobos just yet...

BUT...*Now* D2 has been declared the "main" version of D, suitable and 
recommended for new projects, and is being promoted as such. That changes 
the situation. Our old, previously sensible, approach to Phobos CTFE-ability 
has now become "Breaking regressions in each release."

Therefore, I think *now* is the time when Phobos needs to start having 
regression tests for CTFE-ability. If something doesn't work, then hell, at 
least a quick-and-dirty if(_ctfe) branch would at least be better than 
outright breakage. And if there are worries about that hindering improvement 
of DMD's CTFE mechanism, then a version identifier could be added to turn 
off the CTFE-workaround codepaths.

I realize it's not exactly helpful to say "Hey, you should do X!" instead of 
just simply pitching in and helping. But I'm hoping we can at least agree 
that we've reached a point where it's appropriate, and not premature, for 
this to be done. (One step at a time, right?)
Jul 12 2011
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
I'd like to point out that *normal code* in Phobos is losing
functionality far too often too, like replace() no longer working
on immutable strings as of the last release.

Generalized templates are great, but not at the cost of existing
functionality!
Jul 12 2011
next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On 2011-07-12 15:50, Adam D. Ruppe wrote:
 I'd like to point out that *normal code* in Phobos is losing
 functionality far too often too, like replace() no longer working
 on immutable strings as of the last release.
 
 Generalized templates are great, but not at the cost of existing
 functionality!

Please make sure that you report all such regressions. In some cases, they probably can't be fixed (especially if it's related to a compiler change which fixes something which shouldn't have worked before), but in a lot of such cases, it's likely simply because there wasn't a test case for it, and it wasn't caught. As for immutable arrays, they're generally not going to work with range-based functions because of how templates work ( http://d.puremagic.com/issues/show_bug.cgi?id=6148 ). But if the change requested in http://d.puremagic.com/issues/show_bug.cgi?id=6289 is merged in, then slicing immutable arrays should work. Arguably however, most functions in std.array should work with immutable arrays (since it _is_ std._array_ after all), and if you can assume arrays rather than generic ranges, you can generally make them work with immutable arrays. So, replace may very well be fixed to work with immutable arrays. But if you don't point out such regressions, they don't necessarily get caught. So, please point out such regressions in Phobos. That way they can be fixed and unit tests can be added to help ensure that they don't happen again. - Jonathan M Davis
Jul 12 2011
prev sibling next sibling parent reply bcs <bcs example.com> writes:
== Quote from Adam D. Ruppe (destructionator gmail.com)'s article
 I'd like to point out that *normal code* in Phobos is losing
 functionality far too often too, like replace() no longer working
 on immutable strings as of the last release.
 Generalized templates are great, but not at the cost of existing
 functionality!

The answer to both problems is to have a test suit that is very easy for the average user to add code to that's run along with the current auto-tester. What I'm thinking of is somthing like codepad.org where anyone can submit a code sample and where "interested parties" get informed when the results for them change. With that in place, when some one hits a regression it can be added to the (or a) regression suit with just a few clicks. With a bit more work, you could even let people "go back in time" and show that a test case user to run. Heck, I was looking into building exactly that a few months back but real life got in the way.
Jul 12 2011
parent bcs <bcs google.com> writes:
== Quote from Brad Roberts (braddr puremagic.com)'s article
 On 7/12/2011 7:14 PM, bcs wrote:
 == Quote from Adam D. Ruppe (destructionator gmail.com)'s article
 I'd like to point out that *normal code* in Phobos is losing
 functionality far too often too, like replace() no longer working
 on immutable strings as of the last release.
 Generalized templates are great, but not at the cost of existing
 functionality!

The answer to both problems is to have a test suit that is very


 for the average user to add code to that's run along with the


where
 anyone can submit a code sample and where "interested parties" get


 some one hits a regression it can be added to the (or a)


 suit with just a few clicks. With a bit more work, you could even


 people "go back in time" and show that a test case user to run.

 Heck, I was looking into building exactly that a few months back


 real life got in the way.


Because that requires commit privileges, having git installed and about a dozen other things. I would like to be able to past some code into a web page, tweak it till it shows what I want and post a URL. As for moving stuff from there into phobos; well that might work for selected cases, but for the system I'm thinking of you may end up with a selection of "interesting" cases (ones the devs care about) that's several times the size of phobos and a corpus of "active" test (ones that someone cares about) an order of magnitude larger than that. As long as the search and monitor tools to handle it are done well, the more test the better I say.
Jul 12 2011
prev sibling next sibling parent Brad Roberts <braddr puremagic.com> writes:
On 7/12/2011 7:14 PM, bcs wrote:
 == Quote from Adam D. Ruppe (destructionator gmail.com)'s article
 I'd like to point out that *normal code* in Phobos is losing
 functionality far too often too, like replace() no longer working
 on immutable strings as of the last release.
 Generalized templates are great, but not at the cost of existing
 functionality!

The answer to both problems is to have a test suit that is very easy for the average user to add code to that's run along with the current auto-tester. What I'm thinking of is somthing like codepad.org where anyone can submit a code sample and where "interested parties" get informed when the results for them change. With that in place, when some one hits a regression it can be added to the (or a) regression suit with just a few clicks. With a bit more work, you could even let people "go back in time" and show that a test case user to run. Heck, I was looking into building exactly that a few months back but real life got in the way.

Why separate at all? Add tests to phobos directly like any other unit test.
Jul 12 2011
prev sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Tuesday 12 July 2011 22:50:14 Adam D. Ruppe wrote:
 I'd like to point out that *normal code* in Phobos is losing
 functionality far too often too, like replace() no longer working
 on immutable strings as of the last release.
 
 Generalized templates are great, but not at the cost of existing
 functionality!

https://github.com/D-Programming-Language/phobos/pull/143 - Jonathan M Davis
Jul 12 2011
parent Adam Ruppe <destructionator gmail.com> writes:
 https://github.com/D-Programming-Language/phobos/pull/143

Excellent! Thank you.
Jul 13 2011
prev sibling next sibling parent reply Trass3r <un known.com> writes:
I totally agree.

At least simple functions like string manipulation need to be available at  
compile-time to use string mixins properly.

2.051 broke replace(), then it was fixed.
2.052 broke it again by moving it into std.array and changing the  
implementation.
Now with 2.054 toUpper() is broken.

If you have a library that extensively uses CTFE this is a major problem.
Jul 12 2011
next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
I don't understand what strip() could be doing to break CTFE anyway?
Jul 12 2011
parent reply Johann MacDonagh <johann.macdonagh..no spam..gmail.com> writes:
On 7/12/2011 7:17 PM, Andrej Mitrovic wrote:
 I don't understand what strip() could be doing to break CTFE anyway?

I believe a lot of the std.string functionality was modified to use routines in druntime in this latest release. The default sc.ini links to a static druntime lib as opposed to compiling the source. That means CTFE won't have the source for, in strip's example, _aApplycd2 (rt/aApply.d in druntime). Fixing this (and adding in support for std.intrinsic) should fix a ton of bugs where a "small" change in Phobos kills CTFE.
Jul 12 2011
parent Johann MacDonagh <johann.macdonagh..no spam..gmail.com> writes:
On 7/12/2011 7:35 PM, Trass3r wrote:
 Am 13.07.2011, 01:24 Uhr, schrieb Johann MacDonagh
 <johann.macdonagh..no spam..gmail.com>:

 On 7/12/2011 7:17 PM, Andrej Mitrovic wrote:
 I don't understand what strip() could be doing to break CTFE anyway?

I believe a lot of the std.string functionality was modified to use routines in druntime in this latest release. The default sc.ini links to a static druntime lib as opposed to compiling the source. That means CTFE won't have the source for, in strip's example, _aApplycd2 (rt/aApply.d in druntime). Fixing this (and adding in support for std.intrinsic) should fix a ton of bugs where a "small" change in Phobos kills CTFE.

That's true. I had several of these "no source available" errors. Though I think toUpper crashed at: S toUpper(S)(S s) trusted pure if(isSomeString!S) { foreach (i, dchar cOuter; s) <--- {

That's exactly the issue. When the compiler hits a foreach over a string that wants to convert its character type to another, it has to insert special magic to make that work (UTF decoding, etc...). Clearly this stuff shouldn't be in phobos, so it's in druntime. Unfortunately, the compiler doesn't have access to druntime source by default. I'm not sure how it was doing it before. This worked in 2.053.
Jul 12 2011
prev sibling next sibling parent Trass3r <un known.com> writes:
Am 13.07.2011, 01:24 Uhr, schrieb Johann MacDonagh  
<johann.macdonagh..no spam..gmail.com>:

 On 7/12/2011 7:17 PM, Andrej Mitrovic wrote:
 I don't understand what strip() could be doing to break CTFE anyway?

I believe a lot of the std.string functionality was modified to use routines in druntime in this latest release. The default sc.ini links to a static druntime lib as opposed to compiling the source. That means CTFE won't have the source for, in strip's example, _aApplycd2 (rt/aApply.d in druntime). Fixing this (and adding in support for std.intrinsic) should fix a ton of bugs where a "small" change in Phobos kills CTFE.

That's true. I had several of these "no source available" errors. Though I think toUpper crashed at: S toUpper(S)(S s) trusted pure if(isSomeString!S) { foreach (i, dchar cOuter; s) <--- {
Jul 12 2011
prev sibling parent reply bcs <bcs example.com> writes:
== Quote from Andrej Mitrovic (andrej.mitrovich gmail.com)'s article
 Last time I brought this issue up in bugzilla it was shot down with
 "We don't guarantee and don't have to guarantee functions will

 be CTFE-able between releases".

Maybe there should be a std.ctfe.* that looks a bit like std.* that IS guaranteed to work. Ideally it would be nothing but a bunch of alias.
 I end up having to put initializers in a module ctor. Not nice!

Jul 12 2011
parent reply "Nick Sabalausky" <a a.a> writes:
"bcs" <bcs example.com> wrote in message 
news:iviv9h$2ee$1 digitalmars.com...
 == Quote from Andrej Mitrovic (andrej.mitrovich gmail.com)'s article
 Last time I brought this issue up in bugzilla it was shot down with
 "We don't guarantee and don't have to guarantee functions will

 be CTFE-able between releases".

Maybe there should be a std.ctfe.* that looks a bit like std.* that IS guaranteed to work. Ideally it would be nothing but a bunch of alias.

If we do that we may as well just stick their bodies inside the original function in an if(_ctfe) block.
Jul 12 2011
parent bcs <bcs google.com> writes:
== Quote from Nick Sabalausky (a a.a)'s article
 "bcs" <bcs example.com> wrote in message
 news:iviv9h$2ee$1 digitalmars.com...
 == Quote from Andrej Mitrovic (andrej.mitrovich gmail.com)'s


 Last time I brought this issue up in bugzilla it was shot down



 "We don't guarantee and don't have to guarantee functions will

 be CTFE-able between releases".

Maybe there should be a std.ctfe.* that looks a bit like std.*


 guaranteed to work. Ideally it would be nothing but a bunch of




 function in an if(_ctfe) block.

That only gets a small part of the benefit, the rest is that it would document to the end user what is CTFE clean and also document to the dev what functions need to be CTFE clean. Putting if(_ctfe) in the function ends up making the user look at the code (rather than docs) and still doesn't give a definitive answer as to if the code works this time or, given that, if it may break next time around.
Jul 12 2011
prev sibling next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Last time I brought this issue up in bugzilla it was shot down with
"We don't guarantee and don't have to guarantee functions will always
be CTFE-able between releases".

I end up having to put initializers in a module ctor. Not nice!
Jul 12 2011
parent reply "Nick Sabalausky" <a a.a> writes:
"Andrej Mitrovic" <andrej.mitrovich gmail.com> wrote in message 
news:mailman.1574.1310511524.14074.digitalmars-d puremagic.com...
 Last time I brought this issue up in bugzilla it was shot down with
 "We don't guarantee and don't have to guarantee functions will always
 be CTFE-able between releases".

Yea, I've seen statements to that effect, too. It was a fairly reasonable stance when D2 was the "unstable" branch, but now that D2 is the "main" D, it ultimately means that you can't use Phobos in CTFE because it'll likely break. And that's a *huge* blow to CTFE in general.
Jul 12 2011
parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On 2011-07-12 16:17, Andrej Mitrovic wrote:
 I don't understand what strip() could be doing to break CTFE anyway?

Don has been making huge changes to CTFE. Stuff that didn't used to compile, now complie. Stuff which compiled but shouldn't have now doesn't compile. There's probably stuff which used to compile and should still compile which doesn't compile now too. But with all of those changes, I'm not sure that it's at all reasonable to expect CTFE-ability to be stable. It should be heading in that direction, but I'm not sure how stable Don considers it. Certainly, strip could be failing for a perfectly legitimate reason, or it could be a bug. I have no idea. But with all of the changes that have been being made to CTFE, I'm not at all surprised if stuff has quit working. There's probably more that works now that didn't before, but with all of the recent changes, breakage doesn't surprise me one bit. Having tests in Phobos for CTFE would catch many breakages, but if Don isn't yet guaranteeing what can and can't be CTFEed by the compiler, then such breakages could easily be because of fixes which made it so that stuff which wasn't supposed to compiled but did stopped compiling. So, until Don thinks that what he's been doing to CTFE is appropriately stable and can make guarantees about what will and won't be CTFEable as far as language features go, then Phobos can't make any guarantees about CTFEability. So, basically, a lot of CTFE changes have been happening, and Don has said pretty much said that we're not currently making guarantees about what's CTFEable and what isn't. And until the changes stabilize and Don is willing to make guarantees, Phobos can't guarantee anything about CTFE. - Jonathan M Davis
Jul 12 2011
next sibling parent David Nadlinger <see klickverbot.at> writes:
On 7/13/11 1:29 AM, Jonathan M Davis wrote:
 So, basically, a lot of CTFE changes have been happening, and Don has said
 pretty much said that we're not currently making guarantees about what's
 CTFEable and what isn't. And until the changes stabilize and Don is willing to
 make guarantees, Phobos can't guarantee anything about CTFE.

Yes, but still I think a comprehensive regression test suite would be helpful, also for Don, because it would show him, well, if there are any regressions. I mean, it's not like Don would say, »Hey I just work on CTFE stuff that is fun to do and don't care about the rest«… David
Jul 12 2011
prev sibling next sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Jonathan M Davis" <jmdavisProg gmx.com> wrote in message 
news:mailman.1576.1310513383.14074.digitalmars-d puremagic.com...
 On 2011-07-12 16:17, Andrej Mitrovic wrote:
 I don't understand what strip() could be doing to break CTFE anyway?

Don has been making huge changes to CTFE. Stuff that didn't used to compile, now complie. Stuff which compiled but shouldn't have now doesn't compile. There's probably stuff which used to compile and should still compile which doesn't compile now too. But with all of those changes, I'm not sure that it's at all reasonable to expect CTFE-ability to be stable. It should be heading in that direction, but I'm not sure how stable Don considers it. Certainly, strip could be failing for a perfectly legitimate reason, or it could be a bug. I have no idea. But with all of the changes that have been being made to CTFE, I'm not at all surprised if stuff has quit working. There's probably more that works now that didn't before, but with all of the recent changes, breakage doesn't surprise me one bit.

I definitely expected that in 2.053, since that's the version that had the CTFE overhaul. But it's not being re-overhauled in each version after 2.053 - just bugfixes and added features. So at this point, I think it's reasonable to expect that any CTFE changes that break Phobos code should be fixed before release, even if only a temporary fix, in either the CTFE engine or in the Phobos code that was broken.
Jul 12 2011
parent Don <nospam nospam.com> writes:
Jonathan M Davis wrote:
 On 2011-07-12 17:16, Nick Sabalausky wrote:
 "Jonathan M Davis" <jmdavisProg gmx.com> wrote in message
 news:mailman.1576.1310513383.14074.digitalmars-d puremagic.com...

 On 2011-07-12 16:17, Andrej Mitrovic wrote:
 I don't understand what strip() could be doing to break CTFE anyway?

compile, now complie. Stuff which compiled but shouldn't have now doesn't compile. There's probably stuff which used to compile and should still compile which doesn't compile now too. But with all of those changes, I'm not sure that it's at all reasonable to expect CTFE-ability to be stable. It should be heading in that direction, but I'm not sure how stable Don considers it. Certainly, strip could be failing for a perfectly legitimate reason, or it could be a bug. I have no idea. But with all of the changes that have been being made to CTFE, I'm not at all surprised if stuff has quit working. There's probably more that works now that didn't before, but with all of the recent changes, breakage doesn't surprise me one bit.

CTFE overhaul. But it's not being re-overhauled in each version after 2.053 - just bugfixes and added features. So at this point, I think it's reasonable to expect that any CTFE changes that break Phobos code should be fixed before release, even if only a temporary fix, in either the CTFE engine or in the Phobos code that was broken.

Well, regardless of whether CTFE is still having a major overhaul, there's still plenty of work being done on it. If you look at the changelog, you'll see that there are a lot of CTFE-related changes, and I know that more CTFE changes are in the works for 2.055 based on what Don has said. So, it doesn't surprise me at all that issues with it are cropping up. And unless Don is willing to say that we're going to guarantee the CTFEability of functions, then there will be no guarantees of any functions remaining CTFEable, and we're definitely not going to test that functions stay CTFEable.

2.053 fixed the fundamental wrong-code problem: array references didn't work AT ALL. But it only fixed local variables, not parameters. 2.054 fixed reference parameters. It also completely changed the treatment of slices. After this, most of the wrong-code issues with CTFE were fixed. So that if it compiles in CTFE, it actually works. 2.054 also added support for pointers and heap-allocated structs. This means that CTFE works in enough cases that a large chunk of Phobos should actually work now, and also means that most improvements to Phobos functions shouldn't cause CTFE to fail. My plan is that 2.055 will clean up the smaller issues, which are of two types: (1) special cases that were overlooked (hopefully all of these will cause internal compiler errors rather than wrong-code); and (2) the compiler runtime helper functions. This will leave the CTFE rules as: (1) source code must be available; (2) no access to global or static variables; (3) no asm; (4) no classes (and therefore, no interfaces, can't throw exceptions, etc); (5) no closures. (4) and (5) are only temporary limitations. Note that for any function which is pure, we can expect that non-CTFEable regressions will happen only when external functions are called. Interestingly this is something which could be checked statically. (CTFE can also use functions which aren't pure, but they're considerably more difficult to check, since they are CTFE-callable only for particular combinations of parameters).
Jul 21 2011
prev sibling parent filgood <filgood somewhere.net> writes:
Jonathan,

Given that D2 is now the main version, would it not be possible - maybe 
in the future - if big changes like the CTFE work by Don are planned to 
bring to do this in a 2.1 release of D (and once stable bump to a 2.2 
release or so)... While the big changes happen in 2.1, the 2 releases 
would only hold bug fixes?

~ filgood


On 13/07/2011 00:29, Jonathan M Davis wrote:
 On 2011-07-12 16:17, Andrej Mitrovic wrote:
 I don't understand what strip() could be doing to break CTFE anyway?

Don has been making huge changes to CTFE. Stuff that didn't used to compile, now complie. Stuff which compiled but shouldn't have now doesn't compile. There's probably stuff which used to compile and should still compile which doesn't compile now too. But with all of those changes, I'm not sure that it's at all reasonable to expect CTFE-ability to be stable. It should be heading in that direction, but I'm not sure how stable Don considers it. Certainly, strip could be failing for a perfectly legitimate reason, or it could be a bug. I have no idea. But with all of the changes that have been being made to CTFE, I'm not at all surprised if stuff has quit working. There's probably more that works now that didn't before, but with all of the recent changes, breakage doesn't surprise me one bit. Having tests in Phobos for CTFE would catch many breakages, but if Don isn't yet guaranteeing what can and can't be CTFEed by the compiler, then such breakages could easily be because of fixes which made it so that stuff which wasn't supposed to compiled but did stopped compiling. So, until Don thinks that what he's been doing to CTFE is appropriately stable and can make guarantees about what will and won't be CTFEable as far as language features go, then Phobos can't make any guarantees about CTFEability. So, basically, a lot of CTFE changes have been happening, and Don has said pretty much said that we're not currently making guarantees about what's CTFEable and what isn't. And until the changes stabilize and Don is willing to make guarantees, Phobos can't guarantee anything about CTFE. - Jonathan M Davis

Jul 13 2011
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On 2011-07-12 17:16, Nick Sabalausky wrote:
 "Jonathan M Davis" <jmdavisProg gmx.com> wrote in message
 news:mailman.1576.1310513383.14074.digitalmars-d puremagic.com...
 
 On 2011-07-12 16:17, Andrej Mitrovic wrote:
 I don't understand what strip() could be doing to break CTFE anyway?

Don has been making huge changes to CTFE. Stuff that didn't used to compile, now complie. Stuff which compiled but shouldn't have now doesn't compile. There's probably stuff which used to compile and should still compile which doesn't compile now too. But with all of those changes, I'm not sure that it's at all reasonable to expect CTFE-ability to be stable. It should be heading in that direction, but I'm not sure how stable Don considers it. Certainly, strip could be failing for a perfectly legitimate reason, or it could be a bug. I have no idea. But with all of the changes that have been being made to CTFE, I'm not at all surprised if stuff has quit working. There's probably more that works now that didn't before, but with all of the recent changes, breakage doesn't surprise me one bit.

I definitely expected that in 2.053, since that's the version that had the CTFE overhaul. But it's not being re-overhauled in each version after 2.053 - just bugfixes and added features. So at this point, I think it's reasonable to expect that any CTFE changes that break Phobos code should be fixed before release, even if only a temporary fix, in either the CTFE engine or in the Phobos code that was broken.

Well, regardless of whether CTFE is still having a major overhaul, there's still plenty of work being done on it. If you look at the changelog, you'll see that there are a lot of CTFE-related changes, and I know that more CTFE changes are in the works for 2.055 based on what Don has said. So, it doesn't surprise me at all that issues with it are cropping up. And unless Don is willing to say that we're going to guarantee the CTFEability of functions, then there will be no guarantees of any functions remaining CTFEable, and we're definitely not going to test that functions stay CTFEable. If Don is willing to say that CTFE is stable enough to make guarantees about it and that we can make guarantees about Phobos functions being CTFEable, then I think that it makes perfect sense to add tests for it and make sure that functions remain CTFEable if they're currently CTFEable. But as long as Don is not willing to say that, then Phobos is not going to have any such guarantees and it makes no sense to test for CTFEability. Ideally, I think that we would be able to make such guarantees and test for them, but Don has been against it, and he's the one managing CTFE at this point, so it's really up to him. - Jonathan M Davis
Jul 12 2011
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On 2011-07-13 13:58, filgood wrote:
 Jonathan,
 
 Given that D2 is now the main version, would it not be possible - maybe
 in the future - if big changes like the CTFE work by Don are planned to
 bring to do this in a 2.1 release of D (and once stable bump to a 2.2
 release or so)... While the big changes happen in 2.1, the 2 releases
 would only hold bug fixes?

D versioning doesn't work that way. Currently, we have 1.069 for D1 and 2.054 for D2. The next version of D2 will be 2.055. I don't know exactly what Walter's plans are, but it looks to me like the idea is that 2.100 will follow 2.099. But regardless, D is still in development, and CTFE happens to be one one of its less stable elements. A lot of work is being done on it, and it will probably stabilize relatively soon, but until it does, I don't expect that we'll be making any guarantees about what will or won't be CTFEable. Certainly, Don has been opposed to such suggestions thus far. I hope that once it's more stable, we'll be able to have unit tests in Phobos to guarantee that at least some set of functions stay CTFEable if they're already CTFEable, but we're just not going to make any such guarantees before CTFE is more stable. Not to mention, while most stuff in D is relatively stable, and the language spec really shouldn't be changing much - if at all - at this point, both the compiler and libraries are still under heavy development. Breaking changes should be rare, but they do happen. And newer and/or fancier features are the ones which are most likely to break. CTFE happens to be one of them. There are a couple of others (alias this and inout) which aren't even completely implemented yet. So, it's questionable if talking about D2 as if it were stable is a good idea. It's getting close. IIRC, in a recent discussion on it, Don thought that it would reach that point before the end of the year, but some features just aren't yet stable enough to be guaranteed to continue to work exactly as they do now. Fortunately, that list should be rather short. But CTFE is on it. So, we're not going to fork D2 at this point and have a 2.0 which only has bug fixes and 2.1 which has new features. We're fast approaching the point where what we have will be stable and there shouldn't be any big breaking changes (big breaking changes are actually pretty rare as it is at this point), but we're not going to effectively fork development to provide a more stable branch for a feature for which we're not yet giving guarantees about its stability. I _do_ think that having unit tests in Phobos for CTFEability is a good idea, but until Don says that CTFE is stable enough for that, we're not going to do it. I believe that he's on vacation at the moment (which is presumably why he hasn't said anything on this thread), and maybe he can be persuaded once he's back, but the last time that he had anything to say about it, he was against the idea. - Jonathan M Davis
Jul 13 2011
prev sibling parent reply Don <nospam nospam.com> writes:
Nick Sabalausky wrote:
 Yet again, a new DMD release has broken my code, and other people's code, 
 too, just because of Phobos functions loosing their CTFE-ability. (strip(), 
 toLower(), etc... And yes, I did bring up the strip() regression on the beta 
 list, to no effect.)

The situation is, that prior to 2.054, use of Phobos in CTFE functions has not been supported. No official statements have ever been made that it was supported (and I've tried hard to publicize the fact that it was not supported). There were some fundamental wrong-code bugs and severe limitations in CTFE. And so although some Phobos functions seemed to work in CTFE, most of them were not correct. So, any Phobos function which used to work in CTFE in some old release but doesn't any more is not a regression. It was just dumb luck, relying on undefined behaviour, if any happened to work in a particular release. This has changed with 2.054. Although there are still some restrictions (no classes, many builtin functions aren't implemented), the fundamental wrong code bugs are fixed. So any Phobos function which works in CTFE in 2.054, but fails in a future release, can be considered a regression. We are now in a _completely_ different regime.
Jul 20 2011
next sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On 2011-07-20 15:37, Don wrote:
 Nick Sabalausky wrote:
 Yet again, a new DMD release has broken my code, and other people's code,
 too, just because of Phobos functions loosing their CTFE-ability.
 (strip(), toLower(), etc... And yes, I did bring up the strip()
 regression on the beta list, to no effect.)

The situation is, that prior to 2.054, use of Phobos in CTFE functions has not been supported. No official statements have ever been made that it was supported (and I've tried hard to publicize the fact that it was not supported). There were some fundamental wrong-code bugs and severe limitations in CTFE. And so although some Phobos functions seemed to work in CTFE, most of them were not correct. So, any Phobos function which used to work in CTFE in some old release but doesn't any more is not a regression. It was just dumb luck, relying on undefined behaviour, if any happened to work in a particular release. This has changed with 2.054. Although there are still some restrictions (no classes, many builtin functions aren't implemented), the fundamental wrong code bugs are fixed. So any Phobos function which works in CTFE in 2.054, but fails in a future release, can be considered a regression. We are now in a _completely_ different regime.

If that's the case, then we should look at adding CTFEability tests to at least some portion of Phobos. - Jonathan M Davis
Jul 20 2011
parent "Nick Sabalausky" <a a.a> writes:
"Jonathan M Davis" <jmdavisProg gmx.com> wrote in message 
news:mailman.1817.1311202371.14074.digitalmars-d puremagic.com...
 On 2011-07-20 15:37, Don wrote:
 Nick Sabalausky wrote:
 Yet again, a new DMD release has broken my code, and other people's 
 code,
 too, just because of Phobos functions loosing their CTFE-ability.
 (strip(), toLower(), etc... And yes, I did bring up the strip()
 regression on the beta list, to no effect.)

The situation is, that prior to 2.054, use of Phobos in CTFE functions has not been supported. No official statements have ever been made that it was supported (and I've tried hard to publicize the fact that it was not supported). There were some fundamental wrong-code bugs and severe limitations in CTFE. And so although some Phobos functions seemed to work in CTFE, most of them were not correct. So, any Phobos function which used to work in CTFE in some old release but doesn't any more is not a regression. It was just dumb luck, relying on undefined behaviour, if any happened to work in a particular release.


Ok, I see. I guess I must have missed that.
 This has changed with 2.054. Although there are still some restrictions
 (no classes, many builtin functions aren't implemented), the fundamental
 wrong code bugs are fixed. So any Phobos function which works in CTFE in
 2.054, but fails in a future release, can be considered a regression.
 We are now in a _completely_ different regime.


Glad to hear it!
 If that's the case, then we should look at adding CTFEability tests to at
 least some portion of Phobos.

Totally agree. I think string-processing and anything else likely to be needed in string metaprogramming (such as to!string(int) and to!int(string)) would be the most important ones.
Jul 20 2011
prev sibling next sibling parent reply dsimcha <dsimcha yahoo.com> writes:
Ok, now that the new docs are up, I'm kind of curious about what's a fundamental
limitation vs. a todo.  I assume the following are fundamental limitations,
since
CTFE has no concept of threading or global state and works by interpreting the
code, not executing it directly on the architecture it's being compiled for:

The function source code must be available to the compiler.

Executed expressions may not reference any global or local static variables

No synchronized or asm statements.  (Since CTFE code is always effectively
single-threaded and doesn't allow global state anyhow, could we maybe make
synchronized statements effectively a no-op in CTFE?)

Casts which involve a reinterpretation of the underlying pattern of bits (eg,
from
int[] to float[]) are not permitted

C-style semantics on pointer arithmetic are strictly enforced.

Function parameters may not be C-style variadic


The following look like they are simple matters of programming and could be
implemented eventually:

With statements, scope statements, try-catch-finally statements, throw
statements.

Delete expressions.

Classes and interfaces.


That said, this is a heck of a lot less limitations than CTFE used to have. 
Great
job, Don!

== Quote from Don (nospam nospam.com)'s article
 Nick Sabalausky wrote:
 Yet again, a new DMD release has broken my code, and other people's code,
 too, just because of Phobos functions loosing their CTFE-ability. (strip(),
 toLower(), etc... And yes, I did bring up the strip() regression on the beta
 list, to no effect.)

has not been supported. No official statements have ever been made that it was supported (and I've tried hard to publicize the fact that it was not supported). There were some fundamental wrong-code bugs and severe limitations in CTFE. And so although some Phobos functions seemed to work in CTFE, most of them were not correct. So, any Phobos function which used to work in CTFE in some old release but doesn't any more is not a regression. It was just dumb luck, relying on undefined behaviour, if any happened to work in a particular release. This has changed with 2.054. Although there are still some restrictions (no classes, many builtin functions aren't implemented), the fundamental wrong code bugs are fixed. So any Phobos function which works in CTFE in 2.054, but fails in a future release, can be considered a regression. We are now in a _completely_ different regime.

Jul 26 2011
next sibling parent Don <nospam nospam.com> writes:
dsimcha wrote:
 Ok, now that the new docs are up, I'm kind of curious about what's a
fundamental
 limitation vs. a todo.  I assume the following are fundamental limitations,
since
 CTFE has no concept of threading or global state and works by interpreting the
 code, not executing it directly on the architecture it's being compiled for:
 
 The function source code must be available to the compiler.

Fundamental limitation.
 
 Executed expressions may not reference any global or local static variables

Fundamental limitation.
 No synchronized or asm statements.  (Since CTFE code is always effectively
 single-threaded and doesn't allow global state anyhow, could we maybe make
 synchronized statements effectively a no-op in CTFE?)

Asm fundamental, synchronized could be a no-op.
 
 Casts which involve a reinterpretation of the underlying pattern of bits (eg,
from
 int[] to float[]) are not permitted

Not sure. Would be tricky to implement in general.
 C-style semantics on pointer arithmetic are strictly enforced.

Fundamental, and intentional.
 Function parameters may not be C-style variadic

Not fundamental, this could be implemented fairly easily. But probably shouldn't be encouraged, since it's not necessary.
 The following look like they are simple matters of programming and could be
 implemented eventually:
 
 With statements, scope statements, try-catch-finally statements, throw
statements.
 
 Delete expressions.
 
 Classes and interfaces.

Correct. If you use them, the error message for all of these is "XXX is not yet implemented in CTFE", whereas the others say "cannot be interpreted at compile time". They'll all work eventually. (Not sure about delete, it might not be good idea).
 
 That said, this is a heck of a lot less limitations than CTFE used to have. 
Great
 job, Don!

Thanks. It's been possible to work on this because there are now so many people fixing the critical compiler bugs. In particular, Kennytm has been an enormous help. As well as a phenomenal number of patches, he's also been posting PERFECT bug reports on CTFE issues.
 
 == Quote from Don (nospam nospam.com)'s article
 Nick Sabalausky wrote:
 Yet again, a new DMD release has broken my code, and other people's code,
 too, just because of Phobos functions loosing their CTFE-ability. (strip(),
 toLower(), etc... And yes, I did bring up the strip() regression on the beta
 list, to no effect.)

has not been supported. No official statements have ever been made that it was supported (and I've tried hard to publicize the fact that it was not supported). There were some fundamental wrong-code bugs and severe limitations in CTFE. And so although some Phobos functions seemed to work in CTFE, most of them were not correct. So, any Phobos function which used to work in CTFE in some old release but doesn't any more is not a regression. It was just dumb luck, relying on undefined behaviour, if any happened to work in a particular release. This has changed with 2.054. Although there are still some restrictions (no classes, many builtin functions aren't implemented), the fundamental wrong code bugs are fixed. So any Phobos function which works in CTFE in 2.054, but fails in a future release, can be considered a regression. We are now in a _completely_ different regime.


Jul 30 2011
prev sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday 30 July 2011 21:09:48 Don wrote:
 dsimcha wrote:
 The following look like they are simple matters of programming and could
 be implemented eventually:
 
 With statements, scope statements, try-catch-finally statements, throw
 statements.
 
 Delete expressions.
 
 Classes and interfaces.

Correct. If you use them, the error message for all of these is "XXX is not yet implemented in CTFE", whereas the others say "cannot be interpreted at compile time". They'll all work eventually. (Not sure about delete, it might not be good idea).

Isn't delete supposed to be deprecated? If so, then I definitely wouldn't expect CTFE to support it. - Jonathan M Davis
Jul 30 2011
prev sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
 Ok, now that the new docs are up, I'm kind of curious about what's a
 fundamental limitation vs. a todo. I assume the following are fundamental
 limitations, since CTFE has no concept of threading or global state and
 works by interpreting the code, not executing it directly on the
 architecture it's being compiled for:
 
 The function source code must be available to the compiler.
 
 Executed expressions may not reference any global or local static variables
 
 No synchronized or asm statements. (Since CTFE code is always effectively
 single-threaded and doesn't allow global state anyhow, could we maybe make
 synchronized statements effectively a no-op in CTFE?)
 
 Casts which involve a reinterpretation of the underlying pattern of bits
 (eg, from int[] to float[]) are not permitted
 
 C-style semantics on pointer arithmetic are strictly enforced.
 
 Function parameters may not be C-style variadic
 
 
 The following look like they are simple matters of programming and could be
 implemented eventually:
 
 With statements, scope statements, try-catch-finally statements, throw
 statements.
 
 Delete expressions.
 
 Classes and interfaces.
 
 
 That said, this is a heck of a lot less limitations than CTFE used to have.
 Great job, Don!

Agreed. The situation with regards to CTFE has really been improving. Thanks Don! - Jonathan M Davis
Jul 26 2011