www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - This one weird trick allows you to capture loop variables.

reply Steven Schveighoffer <schveiguy gmail.com> writes:
I thought of this when someone asked the age-old question about 
[closures not capturing loop 
variables](https://issues.dlang.org/show_bug.cgi?id=2043).

https://gist.github.com/schveiguy/b6b037bdfe74997743de81f8d3f4b92b

How does it work? It works because opApply is passed a lambda 
function generated by the compiler to implement the foreach body.

But because this is a *separate* function, when you close over 
that lambda, the lambda's stack is independently allocated on the 
heap for each loop iteration.

This doesn't exactly capture all variables, any variables in the 
enclosing scope are not duplicated (so it doesn't in effect 
capture what values were outside the loop body at that point in 
time). But isn't this the point?

In any case, thought it was interesting, and wondering if 
anyone's every thought of this before.

There are some lifetime and quality of life issues, but it's 
somewhat similar to `std.parallelism` in hooking opApply for 
nifty gain.

-Steve
Oct 19
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
I'm a bit sorry I implemented opApply. The conversion of the loop body to a 
lambda is a tricky beast, and in general such complex rewrites are not a good 
idea. The language should be straightforward.
Oct 19
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, October 19, 2024 8:29:53 PM MDT Walter Bright via Digitalmars-d 
wrote:

 I'm a bit sorry I implemented opApply. The conversion of the loop body to a
 lambda is a tricky beast, and in general such complex rewrites are not a
 good idea. The language should be straightforward.


Honestly, I've always found opApply to be very hard to understand. It's kind
of like it's turning the foreach loop inside out, and it just hurts my
brain. I'd probably understand it much better if I had to use it all the
time, but at this point, I'd only ever use it in cases where ranges didn't
make sense - and even then, I'd probably just elect to use a for loop
instead of trying to take advantage of foreach. foreach is nice, but writing
for loops isn't a big deal IMHO.

But for better or worse, we do have opApply, so I have to deal with it at
least once in a while when someone else has used it. But such is life.

- Jonathan M Davis
Oct 19
parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/19/2024 7:58 PM, Jonathan M Davis wrote:

 it just hurts my brain.


It hurts mine, too.

There's a lot of complex, obtuse code to implement it.
Oct 20
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 10/20/24 04:29, Walter Bright wrote:

 I'm a bit sorry I implemented opApply.


I am not.


 The conversion of the loop body 
 to a lambda is a tricky beast, and in general such complex rewrites are 
 not a good idea. The language should be straightforward.
 


As someone who has also implemented `opApply`, I do think it is quite 
straightforward. It's just internal iteration with support for multiple 
ways to exit the loop, encoded using numerical indices.
Oct 20
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/20/2024 3:45 PM, Timon Gehr wrote:

 As someone who has also implemented `opApply`, I do think it is quite 
 straightforward. It's just internal iteration with support for multiple ways
to 
 exit the loop, encoded using numerical indices.


That's because you're smarter than I am, Timon. And I'm not joking.

I implemented it as a way to deal with visiting each node of a binary tree. 
Doing that with a range is clunky and unappealing.
Oct 20
parent reply =?UTF-8?Q?Ali_=C3=87ehreli?= <acehreli yahoo.com> writes:
On 10/20/24 8:41 PM, Walter Bright wrote:


 That's because you're smarter than I am, Timon. And I'm not joking.


I am happy to be smart enough to see that you are right. :o)


 I implemented it as a way to deal with visiting each node of a binary 
 tree. Doing that with a range is clunky and unappealing.


Exactly! I've always thought opApply is one of the most brilliant parts 
of D. It's easy to see the curly braces of the foreach loop as a lambda 
anyway.

Ali
Nov 19
parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 20/11/2024 5:43 AM, Ali Çehreli wrote:

     I implemented it as a way to deal with visiting each node of a
     binary tree. Doing that with a range is clunky and unappealing.
 
 Exactly! I've always thought opApply is one of the most brilliant parts 
 of D. It's easy to see the curly braces of the foreach loop as a lambda 
 anyway.


There are other ways to do it, than just giving it a delegate.

I.e. hidden state struct and then range over it.

Still on my todo list to look into, as it solves the attribute problem 
of opApply also.
Nov 19
parent reply =?UTF-8?Q?Ali_=C3=87ehreli?= <acehreli yahoo.com> writes:
On 11/19/24 8:50 AM, Richard (Rikki) Andrew Cattermole wrote:

 On 20/11/2024 5:43 AM, Ali Çehreli wrote:



 I've always thought opApply is one of the most brilliant
 parts of D. It's easy to see the curly braces of the foreach loop as a
 lambda anyway.


 There are other ways to do it, than just giving it a delegate.

 I.e. hidden state struct and then range over it.

 Still on my todo list to look into, as it solves the attribute problem
 of opApply also.


Please also keep in mind, as mentioned elsewhere in this thread, e.g. 
tree traversal is trivial with opApply but not with ranges. That's 
because the delegate is there to call regardless of how deeply recursed 
we are in traversal. With ranges, a range must maintain the state of 
traversing.

So, I guess both styles have their strong points.

Ali
Nov 19
parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 20/11/2024 10:43 AM, Ali Çehreli wrote:

 On 11/19/24 8:50 AM, Richard (Rikki) Andrew Cattermole wrote:
  > On 20/11/2024 5:43 AM, Ali Çehreli wrote:
 
  >> I've always thought opApply is one of the most brilliant
  >> parts of D. It's easy to see the curly braces of the foreach loop as a
  >> lambda anyway.
  >
  > There are other ways to do it, than just giving it a delegate.
  >
  > I.e. hidden state struct and then range over it.
  >
  > Still on my todo list to look into, as it solves the attribute problem
  > of opApply also.
 
 Please also keep in mind, as mentioned elsewhere in this thread, e.g. 
 tree traversal is trivial with opApply but not with ranges. That's 
 because the delegate is there to call regardless of how deeply recursed 
 we are in traversal. With ranges, a range must maintain the state of 
 traversing.
 
 So, I guess both styles have their strong points.
 
 Ali


Indeed, its not something I'd consider for normal data structures (I 
think the only place I use them as greedy with recursive calling, is in 
my allocators). It's not a pattern that immediately comes to mind.

In saying that, yeah there are reasons to want opApply in its current 
form, and given the amount of code that uses it already I was never on 
the side of removal :)

In terms of transversal:

```d
struct S {
	DS* ds;
	Node* node;

	ref V front() {
		return node.value;
	}
}

s.front
```

Is no different than:

```d
{
	Node* node;

	del(node.value);
}
```

You have split out the already existing state that's stored in the 
function out of it.

So you can do a ton of what opApply does today with this approach.
Nov 19
prev sibling parent reply Elias (0xEAB) <desisma heidel.beer> writes:
On Sunday, 20 October 2024 at 22:45:24 UTC, Timon Gehr wrote:

 As someone who has also implemented `opApply`, I do think it is 
 quite straightforward. It's just internal iteration with 
 support for multiple ways to exit the loop, encoded using 
 numerical indices.


Do you have any suggestions or insights how we could resolve its 
rather poor compatibility with the attribute soup?
Oct 23
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 10/24/24 01:00, Elias (0xEAB) wrote:

 On Sunday, 20 October 2024 at 22:45:24 UTC, Timon Gehr wrote:

 As someone who has also implemented `opApply`, I do think it is quite 
 straightforward. It's just internal iteration with support for 
 multiple ways to exit the loop, encoded using numerical indices.


 
 Do you have any suggestions or insights how we could resolve its rather 
 poor compatibility with the attribute soup?


Attribute polymorphism. It's a common feature in other effect systems, 
and it is generally useful.

Another solution is to avoid attributes. :)
Oct 23
next sibling parent Max Samukha <maxsamukha gmail.com> writes:
On Thursday, 24 October 2024 at 00:46:48 UTC, Timon Gehr wrote:


 Another solution is to avoid attributes. :)


The only practical one.
Oct 24
prev sibling parent Meta <jared771 gmail.com> writes:
On Thursday, 24 October 2024 at 00:46:48 UTC, Timon Gehr wrote:

 On 10/24/24 01:00, Elias (0xEAB) wrote:

 On Sunday, 20 October 2024 at 22:45:24 UTC, Timon Gehr wrote:

 As someone who has also implemented `opApply`, I do think it 
 is quite straightforward. It's just internal iteration with 
 support for multiple ways to exit the loop, encoded using 
 numerical indices.


 
 Do you have any suggestions or insights how we could resolve 
 its rather poor compatibility with the attribute soup?


 Attribute polymorphism. It's a common feature in other effect 
 systems, and it is generally useful.

 Another solution is to avoid attributes. :)


Do you count languages with effects systems like Koka count as 
having effect polymorphism? I've studied them a bit, and it is a 
deep, deep rabbit hole that D could go down for not that much 
benefit. Or would you limit such a system to something like:

void map(T, U, effect Es...)(t[] a, U function(T) <Es> fun) <Es>; 
// Strawman syntax, I dunno

Where it's solely polymorphism over effects and there isn't all 
this complicated stuff like effect types and handlers, etc.
Oct 24
prev sibling next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Sunday, 20 October 2024 at 02:14:14 UTC, Steven Schveighoffer 
wrote:

 I thought of this when someone asked the age-old question about 
 [closures not capturing loop 
 variables](https://issues.dlang.org/show_bug.cgi?id=2043).

 https://gist.github.com/schveiguy/b6b037bdfe74997743de81f8d3f4b92b

 How does it work? It works because opApply is passed a lambda 
 function generated by the compiler to implement the foreach 
 body.

 But because this is a *separate* function, when you close over 
 that lambda, the lambda's stack is independently allocated on 
 the heap for each loop iteration.


This is cute and all, but the correct solution is to fix the damn 
compiler. The fact that this is necessary in the first place is 
an embarrassment.
Oct 20
next sibling parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Sunday, 20 October 2024 at 17:48:15 UTC, Paul Backus wrote:

 This is cute and all, but the correct solution is to fix the 
 damn compiler. The fact that this is necessary in the first 
 place is an embarrassment.


When I first encountered closures in D and saw how easily they 
capture surrounding variables it felt like magic.

Nowadays I dislike them because they allocate behind the scene, 
and you still get this odd foreach behavior mentioned here - 
which, to be fair, however unintuitive, I believe to be correct 
given the interplay of features.

The only good solution I see is to get rid of implicit captures 
and instead require explicit declaration of what and _how_ to 
capture, i.e. by ref or by value.
Oct 20
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 10/20/24 20:14, Sebastiaan Koppe wrote:

 
 Nowadays I dislike them because they allocate behind the scene, and you 
 still get this odd foreach behavior mentioned here - which, to be fair, 
 however unintuitive, I believe to be correct given the interplay of 
 features.


The odd `foreach` behavior is a long-standing compiler bug.
The behavior with `opApply` is indeed the only correct behavior, no 
interplay of features has to be considered.
Oct 20
prev sibling parent Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Sunday, 20 October 2024 at 17:48:15 UTC, Paul Backus wrote:

 On Sunday, 20 October 2024 at 02:14:14 UTC, Steven 
 Schveighoffer wrote:

 I thought of this when someone asked the age-old question 
 about [closures not capturing loop 
 variables](https://issues.dlang.org/show_bug.cgi?id=2043).

 https://gist.github.com/schveiguy/b6b037bdfe74997743de81f8d3f4b92b

 How does it work? It works because opApply is passed a lambda 
 function generated by the compiler to implement the foreach 
 body.

 But because this is a *separate* function, when you close over 
 that lambda, the lambda's stack is independently allocated on 
 the heap for each loop iteration.


 This is cute and all, but the correct solution is to fix the 
 damn compiler. The fact that this is necessary in the first 
 place is an embarrassment.


Indeed
Nov 21
prev sibling parent Dukc <ajieskola gmail.com> writes:
On Sunday, 20 October 2024 at 02:14:14 UTC, Steven Schveighoffer 
wrote:

 I thought of this when someone asked the age-old question about 
 [closures not capturing loop 
 variables](https://issues.dlang.org/show_bug.cgi?id=2043).

 https://gist.github.com/schveiguy/b6b037bdfe74997743de81f8d3f4b92b


For the most part, I prefer Timon's solution in the Bugzilla 
issue. For this particular example, it would be

```D
void main()
{   import std.stdio;
     void delegate()[] dgs;

     foreach(i; [1, 2, 3, 4])
     {   dgs ~= (x => () => writeln(x))(i);
     }
     foreach(d; dgs) d(); // 1, 2, 3, 4
}
```

I prefer this trick mostly because it also lets you to catch 
other things than just the element being iterated over.
Oct 30