www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Feature request: __traits(canInstantiate), like __traits(compiles) but

reply FeepingCreature <feepingcreature gmail.com> writes:
Right now it's impossible to check "is there a valid (template) 
specialization for this function call" without also saying "and 
by the way, if there's a syntax error in the function, suppress 
the error and return false."

This makes many Phobos and library functions very annoying to use.

A simple (?) fix for this would be adding a new trait that checks 
if the expression resolves to a valid template specialization, 
without attempting to also check if the function body compiles, 
or else without suppressing internal syntax errors.
Jan 15
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Jan 15, 2020 at 03:13:08PM +0000, FeepingCreature via Digitalmars-d
wrote:
 Right now it's impossible to check "is there a valid (template)
 specialization for this function call" without also saying "and by the
 way, if there's a syntax error in the function, suppress the error and
 return false."
 
 This makes many Phobos and library functions very annoying to use.
Yeah, the error-gagging method of speculative template instantiation has been costing me lots of wasted time, too, esp. in Phobos range functions like .find, .filter, etc., because a single typo in the lambda argument causes the template to fail to instantiate, but the only information you get out of it is "cannot instantiate because it must satisfy isInputRange", but the reason it can't satisfy isInputRange is because a syntax error makes the type void (or _error_), but it doesn't tell you WHY the type is void or _error_. Is it a syntax error? A mistyped symbol? A mismatching type? Who knows, you can't tease that information out of the compiler because errors are gagged. And no, turning on verbose output does not help because it floods you with truckloads of "errors" for *other* template instantiations that are completely unrelated to this one (and that actually *worked*, so that's just meaningless noise), and you're left to dig through the mountains of messages to find out which one might actually be relevant to your problem. Furthermore, the error often crops up deep inside a long UFCS chain with multiple levels of nesting, so you can't exactly insert diagnostic messages in the middle to figure out where the problem is. The whole chain is one expression, after all, and all you'll get out of the compiler is that the entire expression failed to compile. But it could be as simple as a single typo deep inside one of the lambdas, or a missing import, and you've no idea where. The only way I could diagnose problems of this sort was to copy-n-paste the entire chain, comment out most of the except the first N steps, and then iteratively uncomment them until it stops compiling. Then once it stops compiling and I identified the offending lambda, copy-n-paste the lambda body into an actual fake function with actual parameter types that the compiler is forced to compile on its own -- because otherwise all errors are gagged and it's impossible to know what the compiler didn't like about it -- just to get the actual error message out of the compiler. And very often, it's a really trivial problem like a typo, a missing import, or forgetting to convert a type to the expected return type, etc.. The sort of error you'd expect the compiler to pinpoint with the exact line number and reason, and that you can fix in 30 seconds. But no thanks to gagged errors, it often takes more like 30 *minutes* just to narrow down the cause of the problem. All in all, it makes for a very poor user experience. No wonder the naysayers have no good things to say about Phobos templates!
 A simple (?) fix for this would be adding a new trait that checks if
 the expression resolves to a valid template specialization, without
 attempting to also check if the function body compiles, or else
 without suppressing internal syntax errors.
+1000. This would've saved me HOURS of debugging trivial problems like missing imports and mistyped identifiers. T -- Chance favours the prepared mind. -- Louis Pasteur
Jan 15
parent reply Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 15 January 2020 at 18:13:08 UTC, H. S. Teoh wrote:
 On Wed, Jan 15, 2020 at 03:13:08PM +0000, FeepingCreature via 
 Digitalmars-d wrote:
 Right now it's impossible to check "is there a valid 
 (template) specialization for this function call" without also 
 saying "and by the way, if there's a syntax error in the 
 function, suppress the error and return false."
 
 This makes many Phobos and library functions very annoying to 
 use.
Yeah, the error-gagging method of speculative template instantiation has been costing me lots of wasted time, too,
I've somehow only just discovered that `-verrors=spec` is a thing that one can pass to dmd. Does that help?
Jan 22
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Jan 22, 2020 at 03:53:15PM +0000, Atila Neves via Digitalmars-d wrote:
 On Wednesday, 15 January 2020 at 18:13:08 UTC, H. S. Teoh wrote:
 On Wed, Jan 15, 2020 at 03:13:08PM +0000, FeepingCreature via
 Digitalmars-d wrote:
 Right now it's impossible to check "is there a valid (template)
 specialization for this function call" without also saying "and by
 the way, if there's a syntax error in the function, suppress the
 error and return false."
 
 This makes many Phobos and library functions very annoying to use.
Yeah, the error-gagging method of speculative template instantiation has been costing me lots of wasted time, too,
I've somehow only just discovered that `-verrors=spec` is a thing that one can pass to dmd. Does that help?
Only barely, because it generally produces pages upon pages of irrelevant errors, and I have to save the output to a file and search for keywords just to find approximately where the error is. And *then* I have to decipher which of the errors are actually relevant. IOW it's a pretty awful user experience. T -- I don't trust computers, I've spent too long programming to think that they can get anything right. -- James Miller
Jan 22
prev sibling next sibling parent MWumpusZ <sinkhole01 bregalad.de> writes:
On Wednesday, 15 January 2020 at 15:13:08 UTC, FeepingCreature 
wrote:
 Right now it's impossible to check "is there a valid (template) 
 specialization for this function call" without also saying "and 
 by the way, if there's a syntax error in the function, suppress 
 the error and return false."
Another example: The case which bothered me, and which I nagged FeepingCreature about leading to the post above, is that he has written us code to automatically JSON-decode to simple D structs, and we can override that automatic decoding where necessary, by writing a T decode(T: SomeSpecialStruct)(const JSONValue value) This works very well.... BUT: If I make an error such that my decode does not even compile, then I get no indication of this - a decoder is automatically generated instead, and at runtime I get strange behavior because the "wrong" decoder is being used. Ideally I get decode errors, but it is easy to imagine cases where that would not even occur.
Jan 16
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/15/20 10:13 AM, FeepingCreature wrote:
 Right now it's impossible to check "is there a valid (template) 
 specialization for this function call" without also saying "and by the 
 way, if there's a syntax error in the function, suppress the error and 
 return false."
I know what you mean, but they are not syntax errors (which ARE errors, even in templates). Which makes them really difficult to categorize.
 This makes many Phobos and library functions very annoying to use.
 
 A simple (?) fix for this would be adding a new trait that checks if the 
 expression resolves to a valid template specialization, without 
 attempting to also check if the function body compiles, or else without 
 suppressing internal syntax errors.
Generally, I use hasMember instead of __triats(compiles) and I'm better off for it. If something defines a member by a specific name, then it better be what I'm looking for, so try using it. If not, then you'll just have to name it something else. I'm not sure Phobos can take this road though. -Steve
Jan 16
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Jan 16, 2020 at 11:00:03AM -0500, Steven Schveighoffer via
Digitalmars-d wrote:
 On 1/15/20 10:13 AM, FeepingCreature wrote:
 Right now it's impossible to check "is there a valid (template)
 specialization for this function call" without also saying "and by
 the way, if there's a syntax error in the function, suppress the
 error and return false."
I know what you mean, but they are not syntax errors (which ARE errors, even in templates). Which makes them really difficult to categorize.
Yeah, they are gagged errors that are nevertheless relevant to the problem at hand. But it's very hard, from the compiler's standpoint, to know which gagged errors are relevant and which are not (most are not, in speculative template instantiations). I'm tempted to propose that anything that passes sig constraints *should* force the compiler to treat any errors in the function body as hard (non-gagged) errors. But then you also have sig constraints that fail due to gagged errors, and it's not clear at all which of those should be treated as hard errors. [...]
 Generally, I use hasMember instead of __triats(compiles) and I'm
 better off for it. If something defines a member by a specific name,
 then it better be what I'm looking for, so try using it. If not, then
 you'll just have to name it something else. I'm not sure Phobos can
 take this road though.
[...] Yeah, over time I've learned to avoid __traits(compiles) as much as possible. It's just far too general, and unhelpful when problems occur. (Why doesn't it compile? It could be any number of reasons, most of which probably the person who wrote the __traits(compiles) didn't even think of.) But anyway, recent versions of dmd are now more helpful, by pointing out which sig constraints are failing, rather than just a blanket "cannot match template" error. The next step IMO is to somehow display a brief summary of why that (or those) particular constraint(s) are failing. It's already quite helpful to know, e.g., that a template function failed to match because isInputRange!R failed. But isInputRange contains quite a number of further constraints; it would be nice to at least have an option to show the dirty details of where that particular constraint failed. I've no idea how to implement such a thing, though. T -- Computerese Irregular Verb Conjugation: I have preferences. You have biases. He/She has prejudices. -- Gene Wirchenko
Jan 16
parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Thursday, 16 January 2020 at 17:30:57 UTC, H. S. Teoh wrote:
 On Thu, Jan 16, 2020 at 11:00:03AM -0500, Steven Schveighoffer 
 via Digitalmars-d wrote:
 On 1/15/20 10:13 AM, FeepingCreature wrote:
 Right now it's impossible to check "is there a valid 
 (template) specialization for this function call" without 
 also saying "and by the way, if there's a syntax error in 
 the function, suppress the error and return false."
I know what you mean, but they are not syntax errors (which ARE errors, even in templates). Which makes them really difficult to categorize.
Yeah, they are gagged errors that are nevertheless relevant to the problem at hand.
I think he means they're semantic errors rather than parser (syntax) errors.
 I'm tempted to propose that anything that passes sig 
 constraints *should* force the compiler to treat any errors in 
 the function body as hard (non-gagged) errors.  But then you 
 also have sig constraints that fail due to gagged errors, and 
 it's not clear at all which of those should be treated as hard 
 errors.
Would be nice, but would be a hard spec change. This is the sort of thing that D should have been doing from the beginning, but we may be too far in now for __traits(compiles) to be changed to do this. However, this is the behavior I would ask for for __traits(canInstantiateWith).
Jan 16
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/17/20 12:54 AM, FeepingCreature wrote:
 
 Would be nice, but would be a hard spec change. This is the sort of 
 thing that D should have been doing from the beginning, but we may be 
 too far in now for __traits(compiles) to be changed to do this. However, 
 this is the behavior I would ask for for __traits(canInstantiateWith).
 
Just to point you at my previous thread on this: https://forum.dlang.org/post/q1jequ$r63$1 digitalmars.com I think more tools for diagnosing why something doesn't happen would be good, but the request of "tell me if I was stupid" is I think impossible for the compiler to figure out. -Steve
Jan 17
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Friday, 17 January 2020 at 14:57:27 UTC, Steven Schveighoffer 
wrote:
 On 1/17/20 12:54 AM, FeepingCreature wrote:
 
 Would be nice, but would be a hard spec change. This is the 
 sort of thing that D should have been doing from the 
 beginning, but we may be too far in now for __traits(compiles) 
 to be changed to do this. However, this is the behavior I 
 would ask for for __traits(canInstantiateWith).
 
Just to point you at my previous thread on this: https://forum.dlang.org/post/q1jequ$r63$1 digitalmars.com I think more tools for diagnosing why something doesn't happen would be good, but the request of "tell me if I was stupid" is I think impossible for the compiler to figure out. -Steve
functions can use only properties of generic types iff such properties were required in the function generic type constraints. AFAIK Nim's Concepts [1] are also moving in that direction, even though their current way is more similar to D (I chatted with one of their core developers, a couple of months ago at a conference, but I may be misremembering some details). In other words: void use(T)(T obj) where (hasMethod!(T, "callMe", int function(string, bool))) { int x = obj.callMe("hello", true); // this works // the following would cause the template definition to // fail to type check (before even being instantiated): // obj.someThingElse = 3; } Of course, requiring complete constraints for D templates would be quite controversial, (although many languages are able to get away with that), but I think it could be a good if we could opt-into that on a per function basis (e.g. using `where`, instead of `if`). If D has such a feature, I believe that libraries like phobos would use it extensively, while user (app) code would use current template duck typing model and only opt-into a stronger model when a developer wants to debug why his template code doesn't work as expected. [1]: https://nim-lang.org/docs/manual_experimental.html#concepts
Jan 17
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/17/20 10:34 AM, Petar Kirov [ZombineDev] wrote:
 On Friday, 17 January 2020 at 14:57:27 UTC, Steven Schveighoffer wrote:
 On 1/17/20 12:54 AM, FeepingCreature wrote:
 Would be nice, but would be a hard spec change. This is the sort of 
 thing that D should have been doing from the beginning, but we may be 
 too far in now for __traits(compiles) to be changed to do this. 
 However, this is the behavior I would ask for for 
 __traits(canInstantiateWith).
Just to point you at my previous thread on this: https://forum.dlang.org/post/q1jequ$r63$1 digitalmars.com I think more tools for diagnosing why something doesn't happen would be good, but the request of "tell me if I was stupid" is I think impossible for the compiler to figure out.
can use only properties of generic types iff such properties were required in the function generic type constraints. AFAIK Nim's Concepts [1] are also moving in that direction, even though their current way is more similar to D (I chatted with one of their core developers, a couple of months ago at a conference, but I may be misremembering some details).
Yes, concepts would be useful, but also would be limited for how D does duck typing. D's constraints are very much like concepts, but without having to declare everything up front for use in the function. But it wouldn't work for UFCS either I don't think. However, the real problem is things like typos that cause the function not to compile for reasons other than the intended failures. For example: T foo(T)(T x) { return X + 5; // typo on parameter name } This will NEVER compile, because X will never exist. The intention was for it to compile with types that support addition to integers (i.e. it's EXPECTED to fail for strings), but instead it fails for all types. The result is that your code selects some other path when you don't want it to. Sometimes this just means it works but is slower, which is even harder to figure out. But it's hard for the compiler to deduce intention from such typos. It would be great if the compiler could figure out this type of error, but I don't think it can. The only "fix" really is to judiciously unittest with the calls you expect to work. And that generally doesn't happen (we are a lazy bunch!). Perhaps, an "expected" clause or something to ensure it compiles for the expected types like: expected(T == int), which wouldn't unittest anything, just make sure it can compile for something that is expected. -Steve
Jan 17
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jan 17, 2020 at 11:01:29AM -0500, Steven Schveighoffer via
Digitalmars-d wrote:
[...]
 Yes, concepts would be useful, but also would be limited for how D
 does duck typing. D's constraints are very much like concepts, but
 without having to declare everything up front for use in the function.
IMO, not declaring everything up front for use is an anti-pattern. Basically, by accepting a template parameter T and performing operations on it, you're assuming the T supports said operations. If T doesn't support such operations, you shouldn't be receiving it in the first place. By not specifying what operations you expect T to support in your sig constraints, you're basically declaring that you accept *any* T, including one that supports no operations, yet you proceed to operate on it, which ought to be an error (not just a silent failure to instantiate). IOW, you're actually making assumptions on what T supports, yet you never declared such assumptions. Conversely, if you declare no assumptions on T, then neither should you be allowed to operate on it. A parameter T received without any assumptions should be treated as an opaque object that allows *no* operations. [...]
 However, the real problem is things like typos that cause the function
 not to compile for reasons other than the intended failures.
 
 For example:
 
 T foo(T)(T x)
 {
    return X + 5; // typo on parameter name
 }
 
 This will NEVER compile, because X will never exist. The intention was
 for it to compile with types that support addition to integers (i.e.
 it's EXPECTED to fail for strings), but instead it fails for all
 types.
See, this is why this kind of code is bad. You basically wish to accept all T that support +, but by not declaring it as such, the compiler has no way to know what you intended. So it assumes that you somehow expect X to spring into existence given some magic value of T, and when that never happens, the compiler always skips over foo. Had you declared that you expect T to support +, then the compiler would know to instantiate foo when T==int, then it would have caught the typo as a compile error. As a bonus, your code would even accept custom types that support +, without any further effort on your part. Another side effect of not declaring assumptions up-front is that the compiler cannot produce better error messages. You're stuck with error gagging that hides typos, because the compiler simply doesn't have enough information to know what was intended. What if you intended for foo never to compile? Without declaring any assumptions the compiler couldn't know better.
 But it's hard for the compiler to deduce intention from such typos. It
 would be great if the compiler could figure out this type of error,
 but I don't think it can.
It can if there was a requirement that template arguments are not allowed to be operated on unless their ability to support the operation is either declared in a sig constraint, or else tested with an appropriate static if condition.
 The only "fix" really is to judiciously unittest with the calls you
 expect to work. And that generally doesn't happen (we are a lazy
 bunch!).
Exactly, that's why you have to force users to declare what assumptions they're making on the template parameters. It's more "convenient" to just take the easy way and allow everything without checking, but it only leads to pain and more pain down the road.
 Perhaps, an "expected" clause or something to ensure it compiles for
 the expected types like: expected(T == int), which wouldn't unittest
 anything, just make sure it can compile for something that is
 expected.
Just do this instead: unittest { // will generate compile error if instantiation fails: alias A = myTemplate!int; } T -- Кто везде - тот нигде.
Jan 17
prev sibling parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Friday, 17 January 2020 at 16:01:29 UTC, Steven Schveighoffer 
wrote:
 However, the real problem is things like typos that cause the 
 function not to compile for reasons other than the intended 
 failures.

 For example:

 T foo(T)(T x)
 {
    return X + 5; // typo on parameter name
 }

 This will NEVER compile, because X will never exist. The 
 intention was for it to compile with types that support 
 addition to integers (i.e. it's EXPECTED to fail for strings), 
 but instead it fails for all types. The result is that your 
 code selects some other path when you don't want it to. 
 Sometimes this just means it works but is slower, which is even 
 harder to figure out.

 But it's hard for the compiler to deduce intention from such 
 typos. It would be great if the compiler could figure out this 
 type of error, but I don't think it can.
Yes it can! D already has support for exactly this kind of thing, with std.traits and `if()` constraints. If you have a metafunction that tries to see if "an alias is usable with string", and it checks that with __traits(canInstantiateCall, foo, string.init) or whatever, then you *should* get a syntax error even in the "return x + 5" case. If you want to avoid that, we don't need language changes, you just need to use the *already existing* mechanism of "if (isNumeric!T)".
Jan 18
next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/18/20 10:21 AM, FeepingCreature wrote:
 On Friday, 17 January 2020 at 16:01:29 UTC, Steven Schveighoffer wrote:
 However, the real problem is things like typos that cause the function 
 not to compile for reasons other than the intended failures.

 For example:

 T foo(T)(T x)
 {
    return X + 5; // typo on parameter name
 }

 This will NEVER compile, because X will never exist. The intention was 
 for it to compile with types that support addition to integers (i.e. 
 it's EXPECTED to fail for strings), but instead it fails for all 
 types. The result is that your code selects some other path when you 
 don't want it to. Sometimes this just means it works but is slower, 
 which is even harder to figure out.

 But it's hard for the compiler to deduce intention from such typos. It 
 would be great if the compiler could figure out this type of error, 
 but I don't think it can.
Yes it can! D already has support for exactly this kind of thing, with std.traits and `if()` constraints. If you have a metafunction that tries to see if "an alias is usable with string", and it checks that with __traits(canInstantiateCall, foo, string.init) or whatever, then you *should* get a syntax error even in the "return x + 5" case. If you want to avoid that, we don't need language changes, you just need to use the *already existing* mechanism of "if (isNumeric!T)".
I don't see how that helps. isNumeric!T is ignoring the problem that X is the wrong symbol name. In the usage of foo, if I say if __traits(compiles, foo(5)) or __traits(canInstantiate, foo, int), how does it know the error during instantiation was not intended? In other words, I need to know, was foo *intended* to compile with T, not *does* it compile with T. -Steve
Jan 21
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/18/20 10:21 AM, FeepingCreature wrote:
 Yes it can!
 
 D already has support for exactly this kind of thing, with std.traits 
 and `if()` constraints. If you have a metafunction that tries to see if 
 "an alias is usable with string", and it checks that with 
 __traits(canInstantiateCall, foo, string.init) or whatever, then you 
 *should* get a syntax error even in the "return x + 5" case. If you want 
 to avoid that, we don't need language changes, you just need to use the 
 *already existing* mechanism of "if (isNumeric!T)".
Wait, I think I get it. You want something that says if the constraints allow the instantiation to proceed, then instantiate, and display the errors as generated. If not, then return false. So to further explore the stupid example I have: T foo(T)(T x) if(isNumeric!T) { return X + 5; } Then __traits(canInstantiate, foo, "") returns false because the constraints are false, and __traits(canInstantiate, foo, 5) produces an error because the implementation is bad. This is definitely possible. But, it could get really annoying. For instance, you may have to repeat the actual implementation in the constraints to get this to match up correctly. e.g.: T foo(T)(T x) if(__traits(compiles, T.init + 5)) { return X + 5; } I mean, isNumeric might not be what you want exactly, for example, you may want foo!BigInt to compile. But, a constraint-free template might be exactly what you wanted to do. Many range types don't care what their element type does, they are just focused on being a container for them. So there's no constraints based on the type. The idea has potential. -Steve
Jan 21
parent user1234 <user1234 1234.de> writes:
On Tuesday, 21 January 2020 at 22:32:45 UTC, Steven Schveighoffer 
wrote:
 On 1/18/20 10:21 AM, FeepingCreature wrote:
 Yes it can!
 
 D already has support for exactly this kind of thing, with 
 std.traits and `if()` constraints. If you have a metafunction 
 that tries to see if "an alias is usable with string", and it 
 checks that with __traits(canInstantiateCall, foo, 
 string.init) or whatever, then you *should* get a syntax error 
 even in the "return x + 5" case. If you want to avoid that, we 
 don't need language changes, you just need to use the *already 
 existing* mechanism of "if (isNumeric!T)".
Wait, I think I get it. You want something that says if the constraints allow the instantiation to proceed, then instantiate, and display the errors as generated. If not, then return false. [...] -Steve
Yes and there's an API for that https://github.com/dlang/dmd/blob/ab2ea3880a144957811d3a8f7c63dfab0e351202/src/dmd/dtemplate.d#L762
Jan 22
prev sibling parent John Colvin <john.loughran.colvin gmail.com> writes:
On Wednesday, 15 January 2020 at 15:13:08 UTC, FeepingCreature 
wrote:
 Right now it's impossible to check "is there a valid (template) 
 specialization for this function call" without also saying "and 
 by the way, if there's a syntax error in the function, suppress 
 the error and return false."

 This makes many Phobos and library functions very annoying to 
 use.

 A simple (?) fix for this would be adding a new trait that 
 checks if the expression resolves to a valid template 
 specialization, without attempting to also check if the 
 function body compiles, or else without suppressing internal 
 syntax errors.
Yes please (although I would say specialization and constraints, for all templates). I worked around this in a commercial project a few years ago by creating an overload set of the passed symbol and a catch-all template, then instantiate and see if I got the catch-all (meaning no match), but obviously this is pretty nasty and doesn't work if one of the overloads of the original template is a catch-all as well.
Jan 17