www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - String lambdas

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
https://github.com/dlang/phobos/pull/3882

I just closed with some regret a nice piece of engineering. Please 
comment if you think string lambdas have a lot of unexploited potential.

One thing we really need in order to 100% replace string lambdas with 
lambdas is function equivalence. Right now we're in the odd situation 
that SomeTemplate!((a, b) => a < b) has distinct types, one per 
instantiation.


Andrei
Apr 26 2016
next sibling parent tsbockman <thomas.bockman gmail.com> writes:
On Tuesday, 26 April 2016 at 17:58:22 UTC, Andrei Alexandrescu 
wrote:

 One thing we really need in order to 100% replace string 
 lambdas with lambdas is function equivalence.


One significant - but undocumented - feature of unaryFun and 
binaryFun, is that they will auto-import the following modules if 
needed:

     import std.traits, std.typecons, std.typetuple;
     import std.algorithm, std.conv, std.exception, std.math, 
std.range, std.string;

Does anyone actually use this feature?
Apr 26 2016
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tuesday, April 26, 2016 13:58:22 Andrei Alexandrescu via Digitalmars-d 
wrote:

 https://github.com/dlang/phobos/pull/3882

 I just closed with some regret a nice piece of engineering. Please
 comment if you think string lambdas have a lot of unexploited potential.


Well, they're nicer than the "new" lambda syntax for really short stuff like
"a == b" or "a != b", but beyond that, the newer syntax mostly does what the
string lambdas were trying to do. And if your lambda gets very long, it
really should be replaced with something like a nested function anyway.
Pretty much anything complicated shouldn't be a lambda, or it risks being
unmaintainable. So, while that PR was indeed a nice piece of engineering, it
seems to be solving a problem that really shouldn't be solved with lambdas
anyway.


 One thing we really need in order to 100% replace string lambdas with
 lambdas is function equivalence. Right now we're in the odd situation
 that SomeTemplate!((a, b) => a < b) has distinct types, one per
 instantiation.


Once that's solved, we can consider deprecating string lambdas, but
equivalence is indeed the one killer feature that non-string lambdas lack.
However, from what I recall of discussions on that, it sounded like it was
going to be pretty nasty to implement. :(

- Jonathan M Davis
Apr 26 2016
prev sibling next sibling parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Tuesday, 26 April 2016 at 17:58:22 UTC, Andrei Alexandrescu 
wrote:

 https://github.com/dlang/phobos/pull/3882

 I just closed with some regret a nice piece of engineering. 
 Please comment if you think string lambdas have a lot of 
 unexploited potential.


I'm of the opinion that string lambdas must go. I started, and I 
really should finish it at some point, removing string lambdas 
from the documentation: https://github.com/dlang/phobos/pull/3800

I think that the drawback you mentioned does not outweigh the 
benefits gained from using actual lambdas.
Apr 26 2016
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 04/26/2016 03:45 PM, Jack Stouffer wrote:

 I think that the drawback you mentioned does not outweigh the benefits
 gained from using actual lambdas.


Actually it turns out to be a major usability issue. -- Andrei
Apr 27 2016
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/27/16 8:31 AM, Andrei Alexandrescu wrote:

 On 04/26/2016 03:45 PM, Jack Stouffer wrote:

 I think that the drawback you mentioned does not outweigh the benefits
 gained from using actual lambdas.


 Actually it turns out to be a major usability issue. -- Andrei


Yes, consider that RedBlackTree!(int, (a, b) => a > b) is going to be a 
different type every time you use it, even if they are 1 line apart!

There are actually 2 things the string lambdas have going for them: 1) 
common instantiation for every usage, and 2) avoiding parentheses with 
instantiation (impossible to do with a short lambda syntax).

I'd still vote for them to go, but we MUST fix the issue with common 
instantiation first.

There has been some discussion in general of using hashing to decrease 
the symbol size for templates, and some discussion about allowing the 
compiler to merge identical binary functions to reduce the size of the 
binaries. Both of those could play in nicely here.

-Steve
Apr 27 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 04/27/2016 11:44 AM, Steven Schveighoffer wrote:

 On 4/27/16 8:31 AM, Andrei Alexandrescu wrote:

 On 04/26/2016 03:45 PM, Jack Stouffer wrote:

 I think that the drawback you mentioned does not outweigh the benefits
 gained from using actual lambdas.


 Actually it turns out to be a major usability issue. -- Andrei


 Yes, consider that RedBlackTree!(int, (a, b) => a > b) is going to be a
 different type every time you use it, even if they are 1 line apart!

 There are actually 2 things the string lambdas have going for them: 1)
 common instantiation for every usage, and 2) avoiding parentheses with
 instantiation (impossible to do with a short lambda syntax).

 I'd still vote for them to go, but we MUST fix the issue with common
 instantiation first.

 There has been some discussion in general of using hashing to decrease
 the symbol size for templates, and some discussion about allowing the
 compiler to merge identical binary functions to reduce the size of the
 binaries. Both of those could play in nicely here.


Yes, you get it exactly right. I think a DIP would be warranted here to 
clarify how lambda equivalence is computed. Could you please draft one? 
-- Andrei
Apr 27 2016
next sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/27/16 1:04 PM, Andrei Alexandrescu wrote:

 On 04/27/2016 11:44 AM, Steven Schveighoffer wrote:

 On 4/27/16 8:31 AM, Andrei Alexandrescu wrote:

 On 04/26/2016 03:45 PM, Jack Stouffer wrote:

 I think that the drawback you mentioned does not outweigh the benefits
 gained from using actual lambdas.


 Actually it turns out to be a major usability issue. -- Andrei


 Yes, consider that RedBlackTree!(int, (a, b) => a > b) is going to be a
 different type every time you use it, even if they are 1 line apart!

 There are actually 2 things the string lambdas have going for them: 1)
 common instantiation for every usage, and 2) avoiding parentheses with
 instantiation (impossible to do with a short lambda syntax).

 I'd still vote for them to go, but we MUST fix the issue with common
 instantiation first.

 There has been some discussion in general of using hashing to decrease
 the symbol size for templates, and some discussion about allowing the
 compiler to merge identical binary functions to reduce the size of the
 binaries. Both of those could play in nicely here.


 Yes, you get it exactly right. I think a DIP would be warranted here to
 clarify how lambda equivalence is computed. Could you please draft one?


I started thinking about this, I'm probably not the best to do this by 
myself, as I'm not familiar with the compiler code. I'm thinking 
possibly we should talk about this next week in person?

BTW, one of the issues with the DIP system I see is that the table for 
DIPs is not auto-generated, so there is a stray dip which hasn't been 
finished and no entry in the table: http://wiki.dlang.org/DIP89

-Steve
Apr 27 2016
prev sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Wednesday, 27 April 2016 at 17:04:47 UTC, Andrei Alexandrescu 
wrote:

 Yes, you get it exactly right. I think a DIP would be warranted 
 here to clarify how lambda equivalence is computed. Could you 
 please draft one? -- Andrei


More generally, it is not clear what is allowed to do for merging 
functions. In C/C++ it is assumed that different function MUST 
have different identities. Namely, if foo and bar MUST have a 
different address.

It means that, even if foo and bar have the same body, you can't 
merge them. Compiler can, however, emit a branch to foo's body 
into bar or vice versa and it is alright.

This is a problem for us if we want to merge templates. I think 
we should abolish this in D, to unlock more merging.
Apr 27 2016
next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 28 April 2016 at 00:14:41 UTC, deadalnix wrote:

 On Wednesday, 27 April 2016 at 17:04:47 UTC, Andrei 
 Alexandrescu wrote:

 Yes, you get it exactly right. I think a DIP would be 
 warranted here to clarify how lambda equivalence is computed. 
 Could you please draft one? -- Andrei


 More generally, it is not clear what is allowed to do for 
 merging functions. In C/C++ it is assumed that different 
 function MUST have different identities. Namely, if foo and bar 
 MUST have a different address.

 It means that, even if foo and bar have the same body, you 
 can't merge them. Compiler can, however, emit a branch to foo's 
 body into bar or vice versa and it is alright.

 This is a problem for us if we want to merge templates. I think 
 we should abolish this in D, to unlock more merging.


Seconded.
Apr 27 2016
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Thursday, 28 April 2016 at 00:14:41 UTC, deadalnix wrote:

 More generally, it is not clear what is allowed to do for 
 merging functions. In C/C++ it is assumed that different 
 function MUST have different identities.


I don't think this needs to hold true for anonymous functions 
though. If they have the same representation twice, it is 
arguably just the same function referenced twice.
Apr 27 2016
parent deadalnix <deadalnix gmail.com> writes:
On Thursday, 28 April 2016 at 03:15:36 UTC, Adam D. Ruppe wrote:

 On Thursday, 28 April 2016 at 00:14:41 UTC, deadalnix wrote:

 More generally, it is not clear what is allowed to do for 
 merging functions. In C/C++ it is assumed that different 
 function MUST have different identities.


 I don't think this needs to hold true for anonymous functions 
 though. If they have the same representation twice, it is 
 arguably just the same function referenced twice.


I'm not enough of a language lawyer to know that. But I can tell 
you that compiler do not take advantage of this at this stage. 
Maybe it is because lambda are fairly new, I'm not sure.
Apr 28 2016
prev sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Wednesday, 27 April 2016 at 12:31:18 UTC, Andrei Alexandrescu 
wrote:

 On 04/26/2016 03:45 PM, Jack Stouffer wrote:

 I think that the drawback you mentioned does not outweigh the 
 benefits
 gained from using actual lambdas.


 Actually it turns out to be a major usability issue. -- Andrei


That would be better for this to go forward in a sensible manner 
that you explains what is the major usability issue here.
Apr 27 2016
parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wednesday, April 27, 2016 21:18:35 deadalnix via Digitalmars-d wrote:

 On Wednesday, 27 April 2016 at 12:31:18 UTC, Andrei Alexandrescu

 wrote:

 On 04/26/2016 03:45 PM, Jack Stouffer wrote:

 I think that the drawback you mentioned does not outweigh the
 benefits
 gained from using actual lambdas.


 Actually it turns out to be a major usability issue. -- Andrei


 That would be better for this to go forward in a sensible manner
 that you explains what is the major usability issue here.


Well, it's problematic with default predicates, because you can't test that
the default is being used (which some algorithms do for efficiency). You're
forced to duplicate the function rather than use a default. But the far
bigger problem is when dealing with types that take a predicate (which is
going to happen with some containers and many ranges). e.g. this compiles

    RedBlackTree!(int, "a < b") a;
    RedBlackTree!(int, "a < b") b = a;

but this does not:

    RedBlackTree!(int, (a, b) => a < b) a;
    RedBlackTree!(int, (a, b) => a < b) b = a;

You get this wonderfully nonsensical error:

q.d(6): Error: cannot implicitly convert expression (a) of type 
q.main.RedBlackTree!(int, (a, b) => a < b, false) to q.main.RedBlackTree!(int, 
(a, b) => a < b, false)

So, you can't convert an expression of one type to an expression of exactly
the same type? Sure, the error message could be improved, but I doubt that
very many folks are going to expect this sort of error unless they've run
into the problem previously. It totally fails to principle of least
surprise. And it's definitely a usability problem.

It makes passing around types with predicates a royal pain when you need to
type them explicitly (which happens far more with containers than ranges).
It also makes having member variables of them harder. To some extent, you
can get around it with typeof(a) - which is often what we have to do with
ranges anyway, but that's pretty ridiculous when you typed out the exact
type to begin with and code-wise, there's no reason why they shouldn't be
identical.

There are likely other uses cases where this is a problem, but those are the
two that we definitely run into with current Phobos code. And while the
problem may make sense to a compiler writer, it's not what much of anyone
else is going to expect or want to deal with.

- Jonathan M Davis
Apr 28 2016
prev sibling parent reply Mint <mintie wute.com> writes:
On Tuesday, 26 April 2016 at 19:45:18 UTC, Jack Stouffer wrote:

 I'm of the opinion that string lambdas must go. I started, and 
 I really should finish it at some point, removing string 
 lambdas from the documentation: 
 https://github.com/dlang/phobos/pull/3800

 I think that the drawback you mentioned does not outweigh the 
 benefits gained from using actual lambdas.


I'm personally somewhat fond of string lambdas for their 
usefulness in making some operations very concise, without 
sacrificing any readability.

ie.

   foo.map!"a.bar".reduce!"a * b";

vs.

   foo.map!(a => a.bar).reduce!((a, b) => a * b);

I'm open to an alternative that is equally short and sweet, but 
replacing them with proper lambda declarations feels like 
uncalled for verbosity to me.
Apr 30 2016
parent reply Seb <seb wilzba.ch> writes:
On Saturday, 30 April 2016 at 15:10:08 UTC, Mint wrote:

 On Tuesday, 26 April 2016 at 19:45:18 UTC, Jack Stouffer wrote:

 I'm of the opinion that string lambdas must go. I started, and 
 I really should finish it at some point, removing string 
 lambdas from the documentation: 
 https://github.com/dlang/phobos/pull/3800

 I think that the drawback you mentioned does not outweigh the 
 benefits gained from using actual lambdas.


 I'm personally somewhat fond of string lambdas for their 
 usefulness in making some operations very concise, without 
 sacrificing any readability.

 ie.

   foo.map!"a.bar".reduce!"a * b";

 vs.

   foo.map!(a => a.bar).reduce!((a, b) => a * b);

 I'm open to an alternative that is equally short and sweet, but 
 replacing them with proper lambda declarations feels like 
 uncalled for verbosity to me.


I really like them too and was impressed that Andrei is so 
against them, especially if you have more arguments writing the 
lambda get's more tedious.
The PR was created to support `each` string lambdas with more 
than two functions, e.g.:

arr.each!`a + 2*b + 4*c + 8*d`
arr.each!((a, b, c, d) = > a + 2*b + 4*c + 8*d)

While I agree that it might be a bit hard to read, they're 
beautifully concise!
Also they are quite popular - even in Phobos code.


 grep -r unaryFun * | wc -l


142


 grep -r binaryFun * | wc -l


250


 grep -r '!".*"' * --exclude datetime.d | wc -l


591 (disclaimer: some false positives)


Btw at mir we have the convention to use backticks, so it's 
easier to distinguish them from normal strings - might be a 
convention that you could adapt or even enforce?
May 01 2016
parent reply Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Sun, 01 May 2016 15:03:41 +0000
Seb via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 I really like them too and was impressed that Andrei is so
 against them, especially if you have more arguments writing the
 lambda get's more tedious.


String lambdas were an innovative solution to make lambdas shorter than
having to provide full-on anonymous functions with brackets and the whole
bit. However, they quickly proved to have problems as soon as you tried to
do much of anything beyond arithmetic and comparisons, because you could
only use functions in them that were in modules imported by std.functional.
So, at some point there was a discussion or two or introducing a shorter
lambda syntax, and we ended up adding the syntax which is currently used

much broader context than string lambdas do. If we'd had it initially, we
probably would never have had string lambdas.

Really, the only three reasons that we still have string lambdas at this
point is to avoid breaking code, because you can't compare lambdas for
equality (whereas you can compare strings), and because the same template
instantiated with the same lambda in multiple places results in different
instantiations, so they're incompatible.

Now, string lambdas are still great for basic stuff like "a != b" and
"a + b" and the like, because even the new lambda syntax is longer than
that. But having the string lambdas on top of the other is adding more
complication to the language/library, and it's going to continue to cause
problems when folks try and use the string lambdas to call functions that
std.functional doesn't have access to - and it wouldn't surprise me if the
recent import fixes actually make it so that std.functional has access to
even fewer functions.

So, on some level it _will_ suck to lose string lambdas even once we've
fixed the issues with non-string lambdas, but they're redundant and will be
potentially confusing to keep around. So, most folks seem to have concluded
that they should be removed (some even seem to think that they should be
removed even without the current issues with non-string lambdas being fixed,
but that's unreasonable IMHO).

- Jonathan M Davis
May 01 2016
parent ag0aep6g <anonymous example.com> writes:
On 02.05.2016 02:17, Jonathan M Davis via Digitalmars-d wrote:

 So, at some point there was a discussion or two or introducing a shorter
 lambda syntax, and we ended up adding the syntax which is currently used



There is a difference. `x => {foo(x);}` has a different meaning in D
May 01 2016
prev sibling next sibling parent QAston <qaston gmail.com> writes:
On Tuesday, 26 April 2016 at 17:58:22 UTC, Andrei Alexandrescu 
wrote:

 One thing we really need in order to 100% replace string 
 lambdas with lambdas is function equivalence. Right now we're 
 in the odd situation that SomeTemplate!((a, b) => a < b) has 
 distinct types, one per instantiation.


Well, SomeTemplate!((a, b) => a < b) uses pass by name, so it's 
sort of expected to be a distinct type for each different object 
you pass to it. Do you want to bind different lambdas (matching 
by some notion of equality) to the same symbol? That'd be an 
enormous hack.

Now, if this was:
SomeTemplate(FunctorType) // type parameter, not alias here
{
this(FunctorType f){
...
}
...
}

and (a, b)=> a < b returned an object of the same type for 
equivalent functions you could have fn equality without giving 
different objects the same symbol. It's also inlineable as pass 
by name is.
Apr 27 2016
prev sibling next sibling parent deadalnix <deadalnix gmail.com> writes:
On Tuesday, 26 April 2016 at 17:58:22 UTC, Andrei Alexandrescu 
wrote:

 https://github.com/dlang/phobos/pull/3882

 I just closed with some regret a nice piece of engineering. 
 Please comment if you think string lambdas have a lot of 
 unexploited potential.

 One thing we really need in order to 100% replace string 
 lambdas with lambdas is function equivalence. Right now we're 
 in the odd situation that SomeTemplate!((a, b) => a < b) has 
 distinct types, one per instantiation.


 Andrei


This can be worked around by defining a less function and using 
it here. This, however, runs into some subtleties because 
functions and functions aren't the same thing in D (Confusing ? 
You bet it is !).
Apr 27 2016
prev sibling next sibling parent reply Bauss <streetzproductionz hotmail.com> writes:
On Tuesday, 26 April 2016 at 17:58:22 UTC, Andrei Alexandrescu 
wrote:

 https://github.com/dlang/phobos/pull/3882

 I just closed with some regret a nice piece of engineering. 
 Please comment if you think string lambdas have a lot of 
 unexploited potential.

 One thing we really need in order to 100% replace string 
 lambdas with lambdas is function equivalence. Right now we're 
 in the odd situation that SomeTemplate!((a, b) => a < b) has 
 distinct types, one per instantiation.


 Andrei


An alternative to string lambads could be something like a 
shortened version of lambdas like consider

"a > b" could be something like {%0 > %1} and then the body of 
the expression is evaluated to see whether it's a valid D 
expression where %1 and %2 would equal first and second arguments 
in the expression and when used you could simply use do something 
like

Compare(T,expression)(T x, T y) {
     return expression(x,y); // the compiler has to figure out 
that this should be evaluated as return x < y with the example 
above.

This will ensure that the expression is a valid D expression and 
cannot really hacked to do other stuff what it's meant to.

Of course this is just an idea on top of my head on how string 
lambdas could be resolved.
Apr 30 2016
parent Bauss <streetzproductionz hotmail.com> writes:
On Saturday, 30 April 2016 at 16:02:02 UTC, Bauss wrote:

 On Tuesday, 26 April 2016 at 17:58:22 UTC, Andrei Alexandrescu 
 wrote:

 [...]


 An alternative to string lambads could be something like a 
 shortened version of lambdas like consider

 [...]


%1 and %2 should be %0 and %1 and also the < should be > in the 
return statement of course.
Apr 30 2016
prev sibling parent reply poliklosio <poliklosio happypizza.com> writes:
On Tuesday, 26 April 2016 at 17:58:22 UTC, Andrei Alexandrescu 
wrote:

 https://github.com/dlang/phobos/pull/3882

 I just closed with some regret a nice piece of engineering. 
 Please comment if you think string lambdas have a lot of 
 unexploited potential.

 One thing we really need in order to 100% replace string 
 lambdas with lambdas is function equivalence. Right now we're 
 in the odd situation that SomeTemplate!((a, b) => a < b) has 
 distinct types, one per instantiation.


 Andrei


I will just point out an obvious solution to the problem of 
having distinct types: if a normal lambda is sufficiently simple, 
and is used as a template parameter, lower the code to the one 
which uses an equivalent string lambda.

The string needs to somewhat normalized so that (a, b) => a < b  
and  (c,d) => c<d  are converted to the same string.

Whether the compiler allows explicit string lambdas in user code 
in another issue.
May 01 2016
parent poliklosio <poliklosio happypizza.com> writes:
 Whether the compiler allows explicit string lambdas in user 
 code in another issue.


s/in another issue/is another issue/
May 01 2016