www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - About foreach loops

reply bearophile <bearophileHUGS lycos.com> writes:
This post is about some characteristics of the D foreach that has nagged me for
some time.

This is a little Python 2.6 program:


for i in xrange(10):
    i += 1
    print i,


Its output shows that you are allowed to modify the iteration variable
(contents of the iteration name), but the iteration goes on with no change:
1 2 3 4 5 6 7 8 9 10


Similar code in D using foreach shows a different story:

import std.stdio;
void main() {
    foreach (i; 0 .. 10) {
        i += 1;
        write(i, " ");
    }
}

The output:
1 3 5 7 9 


In my opinion this is a bit bug-prone because in real code there is some risk
of modifying the iteration variable "i" by mistake. (Note: here I am not
talking about D for() loops. They are OK, their semantics is transparent
enough. foreach() loops are less transparent and they *look* higher level than
for() loops). I'd like the iteration variable to act as being a copy of the
true loop variable as in Python. If this is a bit bad for foreach performance,
then I'd like the compiler to forbid the mutation of the foreach iteration
variable inside the foreach body. Is this even possible?


Currently you can't solve the problem adding a const(int) to the iteration
variable:

import std.stdio;
void main() {
    foreach (const(int) i; 0 .. 10) { // line 3
        write(i, " ");
    }
}


DMD gives:
test.d(3): Error: variable test.main.i cannot modify const


This is a related but different thing:
http://d.puremagic.com/issues/show_bug.cgi?id=5255

Bye,
bearophile
Jun 14 2011
next sibling parent reply Caligo <iteronvexor gmail.com> writes:
I think D is fine and you may be confusing index with element.

The equivalence of your Python example in D is this:

  foreach(e; [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]){
    e += 1;
    write(e, " ");
  }

or this:

  foreach(e; take(recurrence!("a[n]+1")(0), 10)){
    e += 1;
    write(e, " ");
  }

and they both output:
1 2 3 4 5 6 7 8 9 10



This:

  foreach(i; 0..10){
    i += 1;
    write(i, " ");
  }

is the same as this:

  for(int i = 0; i < 10; ++i){
    i += 1;
    write(i, " ");
  }

and they both output:
1 3 5 7 9


This might make it clear:

  foreach(i, e; [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]){
    i += 1;
    writeln(i, " ", e);
  }

which outputs:
1 0
3 2
5 4
7 6
9 8
Jun 14 2011
next sibling parent Jesse Phillips <jessekphillips+D gmail.com> writes:
Nice! I've always expected bearophile's behavior and end up switching to a for
loop when wanting to modify the index.

Caligo Wrote:

 I think D is fine and you may be confusing index with element.
 
 The equivalence of your Python example in D is this:
 
   foreach(e; [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]){
     e += 1;
     write(e, " ");
   }
 
 or this:
 
   foreach(e; take(recurrence!("a[n]+1")(0), 10)){
     e += 1;
     write(e, " ");
   }
 
 and they both output:
 1 2 3 4 5 6 7 8 9 10
 
 
 
 This:
 
   foreach(i; 0..10){
     i += 1;
     write(i, " ");
   }
 
 is the same as this:
 
   for(int i = 0; i < 10; ++i){
     i += 1;
     write(i, " ");
   }
 
 and they both output:
 1 3 5 7 9
 
 
 This might make it clear:
 
   foreach(i, e; [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]){
     i += 1;
     writeln(i, " ", e);
   }
 
 which outputs:
 1 0
 3 2
 5 4
 7 6
 9 8

Jun 14 2011
prev sibling parent bearophile <bearophileHUGS lycos.com> writes:
Caligo:

 I think D is fine and you may be confusing index with element.

Unfortunately I think what's confused here is the design of foreach.
 The equivalence of your Python example in D is this:
 
   foreach(e; [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]){

Nope, xrange is lazy.
   foreach(e; take(recurrence!("a[n]+1")(0), 10)){

There's std.range.iota for this :-)
 This:
 
   foreach(i; 0..10){
     i += 1;
     write(i, " ");
   }
 
 is the same as this:
 
   for(int i = 0; i < 10; ++i){
     i += 1;
     write(i, " ");
   }

The problem is this equivalence is hidden. foreach() loops look higher level than for loops. So programmers expect this higher level look to be associated with a higher level semantics too. This is why I think currently they are a bit bug-prone. Modifying the loop variable of a foreach loop is code smell, if I see it in code I fix it in some way, using a copy of the loop variable, or I replace the foreach loop with a for loop. So I'd like the compiler to "ignore" or probably better refuse such modifications to the foreach loop variable, if possible. Bye, bearophile
Jun 14 2011
prev sibling next sibling parent Ali =?iso-8859-1?q?=C7ehreli?= <acehreli yahoo.com> writes:
On Tue, 14 Jun 2011 23:45:11 -0500, Caligo wrote:

 This might make it clear:
 
   foreach(i, e; [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]){
     i += 1;
     writeln(i, " ", e);
   }
 
 which outputs:
 1 0
 3 2
 5 4
 7 6
 9 8

I think that example strengthens bearophile's point of view. I would expect 'e' to provide access to all of the elements regardless of whether we asked for the extra index information or not. The fact that modifying the extra information changes the elements that get visited is confusing. Despite 'e' being a copy of the element, 'i's being a reference to the actual implementation variable is extra confusing. :) Ali
Jun 14 2011
prev sibling next sibling parent reply KennyTM~ <kennytm gmail.com> writes:
On Jun 15, 11 10:20, bearophile wrote:
 This post is about some characteristics of the D foreach that has nagged me
for some time.

 This is a little Python 2.6 program:


 for i in xrange(10):
      i += 1
      print i,


 Its output shows that you are allowed to modify the iteration variable
(contents of the iteration name), but the iteration goes on with no change:
 1 2 3 4 5 6 7 8 9 10


 Similar code in D using foreach shows a different story:

 import std.stdio;
 void main() {
      foreach (i; 0 .. 10) {
          i += 1;
          write(i, " ");
      }
 }

 The output:
 1 3 5 7 9


 In my opinion this is a bit bug-prone because in real code there is some risk
of modifying the iteration variable "i" by mistake. (Note: here I am not
talking about D for() loops. They are OK, their semantics is transparent
enough. foreach() loops are less transparent and they *look* higher level than
for() loops). I'd like the iteration variable to act as being a copy of the
true loop variable as in Python. If this is a bit bad for foreach performance,
then I'd like the compiler to forbid the mutation of the foreach iteration
variable inside the foreach body. Is this even possible?


 Currently you can't solve the problem adding a const(int) to the iteration
variable:

 import std.stdio;
 void main() {
      foreach (const(int) i; 0 .. 10) { // line 3
          write(i, " ");
      }
 }


 DMD gives:
 test.d(3): Error: variable test.main.i cannot modify const


 This is a related but different thing:
 http://d.puremagic.com/issues/show_bug.cgi?id=5255

 Bye,
 bearophile

Perhaps the `foreach` loop foreach (i; a .. b) loop_body; should be rewritten into for (auto __foreachtemp1234 = a; __foreachtemp1234 < b; ++ __foreachtemp1234) { const i = __foreachtmp1234; loop_body; } this would prevent the loop index be modified inside the loop body. Similarly, the indiced array loop foreach (i, elem; arr) loop_body; should be rewritten into for (size_t __foreachtemp5678 = 0, __foreachtemp5679 = a.length; __foreachtemp5678 < __foreachtemp5679; ++ __foreachtemp5678) { const i = __foreachtemp5678; auto elem = arr[__foreachtemp5678]; loop_body; } bearophile, have you filed an enhancement request for this?
Jun 15 2011
parent bearophile <bearophileHUGS lycos.com> writes:
KennyTM~:

 bearophile, have you filed an enhancement request for this?

I haven't, I'm looking for opinions first, because foreach is a basic D construct. Bye, bearophile
Jun 15 2011
prev sibling next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 14 Jun 2011 22:20:45 -0400, bearophile <bearophileHUGS lycos.com>  
wrote:

 This post is about some characteristics of the D foreach that has nagged  
 me for some time.

 This is a little Python 2.6 program:


 for i in xrange(10):
     i += 1
     print i,


 Its output shows that you are allowed to modify the iteration variable  
 (contents of the iteration name), but the iteration goes on with no  
 change:
 1 2 3 4 5 6 7 8 9 10


 Similar code in D using foreach shows a different story:

 import std.stdio;
 void main() {
     foreach (i; 0 .. 10) {
         i += 1;
         write(i, " ");
     }
 }

 The output:
 1 3 5 7 9

I think this is a bug. Either i should be const during loop iteration, or an additional temporary should be manufactured. It is always possible to get the above behavior using a for loop. foreach should be sane.
 In my opinion this is a bit bug-prone because in real code there is some  
 risk of modifying the iteration variable "i" by mistake. (Note: here I  
 am not talking about D for() loops. They are OK, their semantics is  
 transparent enough. foreach() loops are less transparent and they *look*  
 higher level than for() loops). I'd like the iteration variable to act  
 as being a copy of the true loop variable as in Python. If this is a bit  
 bad for foreach performance, then I'd like the compiler to forbid the  
 mutation of the foreach iteration variable inside the foreach body. Is  
 this even possible?


 Currently you can't solve the problem adding a const(int) to the  
 iteration variable:

 import std.stdio;
 void main() {
     foreach (const(int) i; 0 .. 10) { // line 3
         write(i, " ");
     }
 }


 DMD gives:
 test.d(3): Error: variable test.main.i cannot modify const

generating an additional temporary variable should solve this. Looks like the way to go. -Steve
Jun 15 2011
next sibling parent reply KennyTM~ <kennytm gmail.com> writes:
On Jun 15, 11 22:35, Caligo wrote:
 You can create a temporary if you like:

    foreach(i; 0..10){
      int ii = i + 1;
      writeln(ii, " ");
    }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own.  The following code would not behave correctly:

    foreach(i; 0..10){
      if(i&  1)
        i += 1;
      writeln(i, " is even.");
    }

 not to mention all the code that's already using foreach.

If the code rely on the 'implicit ref' behavior of 'i', the code is doing it wrong. The spec never mentioned what happened when the variable is modified in a ForeachRangeStatement. I believe it is more sane if modifying 'i' is disallowed unless 'ref' is used.
Jun 15 2011
next sibling parent reply KennyTM~ <kennytm gmail.com> writes:
On Jun 15, 11 23:23, Caligo wrote:
 On Wed, Jun 15, 2011 at 9:44 AM, KennyTM~<kennytm gmail.com>  wrote:
 On Jun 15, 11 22:35, Caligo wrote:
 You can create a temporary if you like:

    foreach(i; 0..10){
      int ii = i + 1;
      writeln(ii, " ");
    }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own.  The following code would not behave correctly:

    foreach(i; 0..10){
      if(i&    1)
        i += 1;
      writeln(i, " is even.");
    }

 not to mention all the code that's already using foreach.

If the code rely on the 'implicit ref' behavior of 'i', the code is doing it wrong. The spec never mentioned what happened when the variable is modified in a ForeachRangeStatement. I believe it is more sane if modifying 'i' is disallowed unless 'ref' is used.

This should be a compile time error: foreach(i; 0..10){ i += 1; write(i, " "); } This should be legal. foreach(ref i; 0..10){ i += 1; write(i, " "); } Is that the general consensus here?

I'm +1 on this.
Jun 15 2011
parent Daniel Gibson <metalcaedes gmail.com> writes:
Am 15.06.2011 17:29, schrieb KennyTM~:
 On Jun 15, 11 23:23, Caligo wrote:
 On Wed, Jun 15, 2011 at 9:44 AM, KennyTM~<kennytm gmail.com>  wrote:
 On Jun 15, 11 22:35, Caligo wrote:
 You can create a temporary if you like:

    foreach(i; 0..10){
      int ii = i + 1;
      writeln(ii, " ");
    }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own.  The following code would not behave correctly:

    foreach(i; 0..10){
      if(i&    1)
        i += 1;
      writeln(i, " is even.");
    }

 not to mention all the code that's already using foreach.

If the code rely on the 'implicit ref' behavior of 'i', the code is doing it wrong. The spec never mentioned what happened when the variable is modified in a ForeachRangeStatement. I believe it is more sane if modifying 'i' is disallowed unless 'ref' is used.

This should be a compile time error: foreach(i; 0..10){ i += 1; write(i, " "); } This should be legal. foreach(ref i; 0..10){ i += 1; write(i, " "); } Is that the general consensus here?

I'm +1 on this.

Actually I thought it'd already be like this (or i being a copy). So +1 from me as well.
Jun 15 2011
prev sibling parent reply KennyTM~ <kennytm gmail.com> writes:
On Jun 15, 11 23:23, Caligo wrote:
 On Wed, Jun 15, 2011 at 9:44 AM, KennyTM~<kennytm gmail.com>  wrote:
 On Jun 15, 11 22:35, Caligo wrote:
 You can create a temporary if you like:

    foreach(i; 0..10){
      int ii = i + 1;
      writeln(ii, " ");
    }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own.  The following code would not behave correctly:

    foreach(i; 0..10){
      if(i&    1)
        i += 1;
      writeln(i, " is even.");
    }

 not to mention all the code that's already using foreach.

If the code rely on the 'implicit ref' behavior of 'i', the code is doing it wrong. The spec never mentioned what happened when the variable is modified in a ForeachRangeStatement. I believe it is more sane if modifying 'i' is disallowed unless 'ref' is used.

This should be a compile time error: foreach(i; 0..10){ i += 1; write(i, " "); } This should be legal. foreach(ref i; 0..10){ i += 1; write(i, " "); } Is that the general consensus here?

A simple patch for this: https://github.com/kennytm/dmd/compare/master...const_foreach_range
Jun 15 2011
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/15/11 11:51 AM, KennyTM~ wrote:
 On Jun 15, 11 23:23, Caligo wrote:
 On Wed, Jun 15, 2011 at 9:44 AM, KennyTM~<kennytm gmail.com> wrote:
 On Jun 15, 11 22:35, Caligo wrote:
 You can create a temporary if you like:

 foreach(i; 0..10){
 int ii = i + 1;
 writeln(ii, " ");
 }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own. The following code would not behave correctly:

 foreach(i; 0..10){
 if(i& 1)
 i += 1;
 writeln(i, " is even.");
 }

 not to mention all the code that's already using foreach.

If the code rely on the 'implicit ref' behavior of 'i', the code is doing it wrong. The spec never mentioned what happened when the variable is modified in a ForeachRangeStatement. I believe it is more sane if modifying 'i' is disallowed unless 'ref' is used.

This should be a compile time error: foreach(i; 0..10){ i += 1; write(i, " "); } This should be legal. foreach(ref i; 0..10){ i += 1; write(i, " "); } Is that the general consensus here?

A simple patch for this: https://github.com/kennytm/dmd/compare/master...const_foreach_range

Does that take care of change through an alias, e.g. int * p = i; ++*p; Andrei
Jun 15 2011
parent reply KennyTM~ <kennytm gmail.com> writes:
On Jun 16, 11 00:53, Andrei Alexandrescu wrote:
 On 6/15/11 11:51 AM, KennyTM~ wrote:
 On Jun 15, 11 23:23, Caligo wrote:
 On Wed, Jun 15, 2011 at 9:44 AM, KennyTM~<kennytm gmail.com> wrote:
 On Jun 15, 11 22:35, Caligo wrote:
 You can create a temporary if you like:

 foreach(i; 0..10){
 int ii = i + 1;
 writeln(ii, " ");
 }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own. The following code would not behave correctly:

 foreach(i; 0..10){
 if(i& 1)
 i += 1;
 writeln(i, " is even.");
 }

 not to mention all the code that's already using foreach.

If the code rely on the 'implicit ref' behavior of 'i', the code is doing it wrong. The spec never mentioned what happened when the variable is modified in a ForeachRangeStatement. I believe it is more sane if modifying 'i' is disallowed unless 'ref' is used.

This should be a compile time error: foreach(i; 0..10){ i += 1; write(i, " "); } This should be legal. foreach(ref i; 0..10){ i += 1; write(i, " "); } Is that the general consensus here?

A simple patch for this: https://github.com/kennytm/dmd/compare/master...const_foreach_range

Does that take care of change through an alias, e.g. int * p = i; ++*p; Andrei

Error: cannot implicitly convert expression (i) of type const(int) to int* What the patch does is simply rewrite foreach (key; lwr .. upr) body; into foreach (auto __iterator1234 = lwr, auto __limit1235 = upr; __iterator1234 < __limit1235; __iterator1234++) { const key = __iterator1234; body; } so as long as the 'const' system in D has no bug, the compiler can deny all unintended mutation in the loop.
Jun 15 2011
next sibling parent KennyTM~ <kennytm gmail.com> writes:
On Jun 16, 11 02:14, KennyTM~ wrote:
 On Jun 16, 11 00:53, Andrei Alexandrescu wrote:
 On 6/15/11 11:51 AM, KennyTM~ wrote:
 On Jun 15, 11 23:23, Caligo wrote:
 On Wed, Jun 15, 2011 at 9:44 AM, KennyTM~<kennytm gmail.com> wrote:
 On Jun 15, 11 22:35, Caligo wrote:
 You can create a temporary if you like:

 foreach(i; 0..10){
 int ii = i + 1;
 writeln(ii, " ");
 }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own. The following code would not behave correctly:

 foreach(i; 0..10){
 if(i& 1)
 i += 1;
 writeln(i, " is even.");
 }

 not to mention all the code that's already using foreach.

If the code rely on the 'implicit ref' behavior of 'i', the code is doing it wrong. The spec never mentioned what happened when the variable is modified in a ForeachRangeStatement. I believe it is more sane if modifying 'i' is disallowed unless 'ref' is used.

This should be a compile time error: foreach(i; 0..10){ i += 1; write(i, " "); } This should be legal. foreach(ref i; 0..10){ i += 1; write(i, " "); } Is that the general consensus here?

A simple patch for this: https://github.com/kennytm/dmd/compare/master...const_foreach_range

Does that take care of change through an alias, e.g. int * p = i; ++*p; Andrei

Error: cannot implicitly convert expression (i) of type const(int) to int* What the patch does is simply rewrite foreach (key; lwr .. upr) body; into foreach (auto __iterator1234 = lwr, auto __limit1235 = upr; __iterator1234 < __limit1235; __iterator1234++) { const key = __iterator1234; body; } so as long as the 'const' system in D has no bug, the compiler can deny all unintended mutation in the loop.

Actually I think you meant 'int* p = &i;'. Well that is detected too. Error: cannot implicitly convert expression (& i) of type const(int)* to int*
Jun 15 2011
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/15/11 1:14 PM, KennyTM~ wrote:
 On Jun 16, 11 00:53, Andrei Alexandrescu wrote:
 On 6/15/11 11:51 AM, KennyTM~ wrote:
 On Jun 15, 11 23:23, Caligo wrote:
 On Wed, Jun 15, 2011 at 9:44 AM, KennyTM~<kennytm gmail.com> wrote:
 On Jun 15, 11 22:35, Caligo wrote:
 You can create a temporary if you like:

 foreach(i; 0..10){
 int ii = i + 1;
 writeln(ii, " ");
 }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own. The following code would not behave correctly:

 foreach(i; 0..10){
 if(i& 1)
 i += 1;
 writeln(i, " is even.");
 }

 not to mention all the code that's already using foreach.

If the code rely on the 'implicit ref' behavior of 'i', the code is doing it wrong. The spec never mentioned what happened when the variable is modified in a ForeachRangeStatement. I believe it is more sane if modifying 'i' is disallowed unless 'ref' is used.

This should be a compile time error: foreach(i; 0..10){ i += 1; write(i, " "); } This should be legal. foreach(ref i; 0..10){ i += 1; write(i, " "); } Is that the general consensus here?

A simple patch for this: https://github.com/kennytm/dmd/compare/master...const_foreach_range

Does that take care of change through an alias, e.g. int * p = i; ++*p; Andrei

Error: cannot implicitly convert expression (i) of type const(int) to int* What the patch does is simply rewrite foreach (key; lwr .. upr) body; into foreach (auto __iterator1234 = lwr, auto __limit1235 = upr; __iterator1234 < __limit1235; __iterator1234++) { const key = __iterator1234; body; } so as long as the 'const' system in D has no bug, the compiler can deny all unintended mutation in the loop.

I understand, thanks. One issue we need to be careful about is performance. You need to ensure that the code ultimately produced is not impacted at all. Also - for now please limit this to built-in numbers only (it won't work for pointers), and replace const with immutable. Thanks, Andrei
Jun 15 2011
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 6/15/11 1:14 PM, KennyTM~ wrote:
 On Jun 16, 11 00:53, Andrei Alexandrescu wrote:
 On 6/15/11 11:51 AM, KennyTM~ wrote:
 On Jun 15, 11 23:23, Caligo wrote:
 On Wed, Jun 15, 2011 at 9:44 AM, KennyTM~<kennytm gmail.com> wrote:
 On Jun 15, 11 22:35, Caligo wrote:
 You can create a temporary if you like:

 foreach(i; 0..10){
 int ii = i + 1;
 writeln(ii, " ");
 }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own. The following code would not behave correctly:

 foreach(i; 0..10){
 if(i& 1)
 i += 1;
 writeln(i, " is even.");
 }

 not to mention all the code that's already using foreach.

If the code rely on the 'implicit ref' behavior of 'i', the code is doing it wrong. The spec never mentioned what happened when the variable is modified in a ForeachRangeStatement. I believe it is more sane if modifying 'i' is disallowed unless 'ref' is used.

This should be a compile time error: foreach(i; 0..10){ i += 1; write(i, " "); } This should be legal. foreach(ref i; 0..10){ i += 1; write(i, " "); } Is that the general consensus here?

A simple patch for this: https://github.com/kennytm/dmd/compare/master...const_foreach_range

Does that take care of change through an alias, e.g. int * p = i; ++*p; Andrei

Error: cannot implicitly convert expression (i) of type const(int) to int* What the patch does is simply rewrite foreach (key; lwr .. upr) body; into foreach (auto __iterator1234 = lwr, auto __limit1235 = upr; __iterator1234 < __limit1235; __iterator1234++) { const key = __iterator1234; body; } so as long as the 'const' system in D has no bug, the compiler can deny all unintended mutation in the loop.

key should not be const imho. 1. const pollution. 2. does not work satisfactory with pointers/user defined structs with indirections. 3? Does not work with structs with a postblit: tt.d(16): Error: function tt.S.__cpctor (ref S p) is not callable using argument types (S) const Is that a compiler bug or a limitation of the language? Is there any way to define a const postblit operator? If yes, see 1. Timon
Jun 15 2011
parent KennyTM~ <kennytm gmail.com> writes:
On Jun 16, 11 03:20, Timon Gehr wrote:
 On 6/15/11 1:14 PM, KennyTM~ wrote:
 On Jun 16, 11 00:53, Andrei Alexandrescu wrote:
 On 6/15/11 11:51 AM, KennyTM~ wrote:
 On Jun 15, 11 23:23, Caligo wrote:
 On Wed, Jun 15, 2011 at 9:44 AM, KennyTM~<kennytm gmail.com>  wrote:
 On Jun 15, 11 22:35, Caligo wrote:
 You can create a temporary if you like:

 foreach(i; 0..10){
 int ii = i + 1;
 writeln(ii, " ");
 }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own. The following code would not behave correctly:

 foreach(i; 0..10){
 if(i&  1)
 i += 1;
 writeln(i, " is even.");
 }

 not to mention all the code that's already using foreach.

If the code rely on the 'implicit ref' behavior of 'i', the code is doing it wrong. The spec never mentioned what happened when the variable is modified in a ForeachRangeStatement. I believe it is more sane if modifying 'i' is disallowed unless 'ref' is used.

This should be a compile time error: foreach(i; 0..10){ i += 1; write(i, " "); } This should be legal. foreach(ref i; 0..10){ i += 1; write(i, " "); } Is that the general consensus here?

A simple patch for this: https://github.com/kennytm/dmd/compare/master...const_foreach_range

Does that take care of change through an alias, e.g. int * p = i; ++*p; Andrei

Error: cannot implicitly convert expression (i) of type const(int) to int* What the patch does is simply rewrite foreach (key; lwr .. upr) body; into foreach (auto __iterator1234 = lwr, auto __limit1235 = upr; __iterator1234< __limit1235; __iterator1234++) { const key = __iterator1234; body; } so as long as the 'const' system in D has no bug, the compiler can deny all unintended mutation in the loop.

key should not be const imho.

Then the compiler will not complain about mutating the copy of index. This may silently break code that rely on the 'ref' behavior (there are 2 foreach loops in Phobos that does this, one in std.path and another in std.xml). This can be a good or bad thing. I'm agnostic on this.
 1. const pollution.

?
 2. does not work satisfactory with pointers/user defined structs with
indirections.

Yes.
 3? Does not work with structs with a postblit:
 tt.d(16): Error: function tt.S.__cpctor (ref S p) is not callable using
argument
 types (S) const
 Is that a compiler bug or a limitation of the language? Is there any way to
define
 a const postblit operator? If yes, see 1.

http://d.puremagic.com/issues/show_bug.cgi?id=4867
 Timon

Jun 15 2011
prev sibling parent reply Don <nospam nospam.com> writes:
KennyTM~ wrote:
 On Jun 16, 11 00:53, Andrei Alexandrescu wrote:
 On 6/15/11 11:51 AM, KennyTM~ wrote:
 On Jun 15, 11 23:23, Caligo wrote:
 On Wed, Jun 15, 2011 at 9:44 AM, KennyTM~<kennytm gmail.com> wrote:
 On Jun 15, 11 22:35, Caligo wrote:
 You can create a temporary if you like:

 foreach(i; 0..10){
 int ii = i + 1;
 writeln(ii, " ");
 }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own. The following code would not behave correctly:

 foreach(i; 0..10){
 if(i& 1)
 i += 1;
 writeln(i, " is even.");
 }

 not to mention all the code that's already using foreach.

If the code rely on the 'implicit ref' behavior of 'i', the code is doing it wrong. The spec never mentioned what happened when the variable is modified in a ForeachRangeStatement. I believe it is more sane if modifying 'i' is disallowed unless 'ref' is used.

This should be a compile time error: foreach(i; 0..10){ i += 1; write(i, " "); } This should be legal. foreach(ref i; 0..10){ i += 1; write(i, " "); } Is that the general consensus here?

A simple patch for this: https://github.com/kennytm/dmd/compare/master...const_foreach_range

Does that take care of change through an alias, e.g. int * p = i; ++*p; Andrei

Error: cannot implicitly convert expression (i) of type const(int) to int* What the patch does is simply rewrite foreach (key; lwr .. upr) body; into foreach (auto __iterator1234 = lwr, auto __limit1235 = upr; __iterator1234 < __limit1235; __iterator1234++) { const key = __iterator1234; body; } so as long as the 'const' system in D has no bug, the compiler can deny all unintended mutation in the loop.

The alternative would be to create __iterator1234 as const, and then cast away const in the ++ step. Ugly, but would eliminate the performance cost for structs.
Jun 15 2011
parent reply bearophile <bearophileHUGS lycos.com> writes:
Don:

 The alternative would be to create __iterator1234 as const, and then 
 cast away const in the ++ step. Ugly, but would eliminate the 
 performance cost for structs.

Isn't it undefined in D to modify a const? And Andrei suggests built-ins only: http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=138718 (I think for the x..y foreach range syntax only.) Bye, bearophile
Jun 15 2011
next sibling parent Don <nospam nospam.com> writes:
bearophile wrote:
 Don:
 
 The alternative would be to create __iterator1234 as const, and then 
 cast away const in the ++ step. Ugly, but would eliminate the 
 performance cost for structs.

Isn't it undefined in D to modify a const?

Yes, but that's in the user world. This is in the compiler world. Lowering frequently creates constructs which aren't well-formed D code. For example, the compiler creates variables in comma expressions, which isn't legal D (it won't even compile).
Jun 17 2011
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
Jesse Phillips wrote:
 On Wed, 15 Jun 2011 16:40:24 -0400, bearophile wrote:

 Isn't it undefined in D to modify a const?

Yes, but unlike what every redditer is say, that doesn't mean the compiler can do whatever it wants. It means you can't define its behavior. Go ahead try. [snip.]

How do "cannot do what it wants" and "the behavior cannot be defined" not contradict each other? It may work as you would expect in some cases but not in others. "The compiler can do what it wants". The behavior will depend on what the compiler wants. The way out would be to just define the behavior of casting away const and mutating the memory to do the expected thing in case it is not typed as immutable somewhere in the program. Would that kill any opportunities for optimization? I don't think so. Timon
Jun 17 2011
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
Jesse Phillips wrote:
 On Wed, 15 Jun 2011 16:40:24 -0400, bearophile wrote:

 Isn't it undefined in D to modify a const?

Yes, but unlike what every redditer is say, that doesn't mean the compiler can do whatever it wants. It means you can't define its behavior. Go ahead try. [snip.]

I tried: void main(){ const c=128; auto p = &c; *cast(int*)p = 64; assert(p==&c); assert(*p!=c); //dmd 2.053: PASS } void main(){ for(const i=0;i<32;++*cast(int*)&i){} assert(0); // 'pass' ;) } Another compiler might decide to fail both assertions or do some funny stuff. Timon
Jun 17 2011
prev sibling parent reply Jesse Phillips <jessekphillips+D gmail.com> writes:
Timon Gehr Wrote:

 Jesse Phillips wrote:
 On Wed, 15 Jun 2011 16:40:24 -0400, bearophile wrote:

 Isn't it undefined in D to modify a const?

Yes, but unlike what every redditer is say, that doesn't mean the compiler can do whatever it wants. It means you can't define its behavior. Go ahead try. [snip.]

How do "cannot do what it wants" and "the behavior cannot be defined" not contradict each other?

Because the compiler can't identify when it happens. Try making the compiler format your hard drive when you modify const. It can't do what it wants because it can't define what it will do!
 It may work as you would expect in some cases but not in others. "The compiler
can
 do what it wants". The behavior will depend on what the compiler wants. The way
 out would be to just define the behavior of casting away const

Casting away const is defined. But once you do, the compiler can no longer identify when you are modifying its value. void main(){ const a = 53; auto pa = &a; auto pb = cast(int*) pa; assert(*pb == *pa); // Always true, all implementations }
 and mutating the
 memory to do the expected thing in case it is not typed as immutable somewhere
in
 the program.

This is not possible. You don't know where the variable resides at compile time. The compiler doesn't know where in memory the values are stored and it isn't allowed to add code which checks where the variable is stored. This means if the variable is in modifiable memory then when you modify it, it will do exactly as you expect, even if it comes from an immutable declaration. Your examples are flawed, declaring a variable with const is equivalent to declaring it immutable, the compiler is allowed to place it in read only memory (or at least that is what most people expect, I don't really know if the spec says anything on it). void main(){ int a = 53; const int* pa = &a; auto pb = cast(int*) pa; assert(*pb == *pa); *pb = 3; assert(*pb == *pa); } This will always be true on every implementation.
Jun 17 2011
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
Jesse Phillips wrote:
 Timon Gehr Wrote:

 Jesse Phillips wrote:
 On Wed, 15 Jun 2011 16:40:24 -0400, bearophile wrote:

 Isn't it undefined in D to modify a const?

Yes, but unlike what every redditer is say, that doesn't mean the compiler can do whatever it wants. It means you can't define its behavior. Go ahead try. [snip.]

How do "cannot do what it wants" and "the behavior cannot be defined" not contradict each other?

Because the compiler can't identify when it happens. Try making the compiler

can't define what it will do! Uh, you could modify DMD to do just that... Left as an exercise. The point is, the resulting compiler would be compliant with the D spec. Ergo a D compiler can do just what it wants in case it has compile code with undefined behavior. And how "can't [the compiler] define what it will do"? It is generating the resulting executable after all. The compiler is *omnipotent*!
 It may work as you would expect in some cases but not in others. "The compiler
can
 do what it wants". The behavior will depend on what the compiler wants. The way
 out would be to just define the behavior of casting away const

Casting away const is defined. But once you do, the compiler can no longer

 void main(){
     const a = 53;
     auto pa = &a;
     auto pb = cast(int*) pa;
     assert(*pb == *pa); // Always true, all implementations

 and mutating the
 memory to do the expected thing in case it is not typed as immutable somewhere
in
 the program.

This is not possible. You don't know where the variable resides

I will now do the same thing you did: Yes I know where it resides, I can disassemble the resulting executable and there is the address, plain as day! JK, I guess you did not realize there was a "...and mutating the memory" part in my statement? =)
 at compile time. The compiler doesn't know where in memory the values are
stored

 code which checks where the variable is stored. This means if the variable is
in

if it comes > from an immutable declaration. I think you did not understand the point. The compiler does not have to check anything (and why do you think it is not allowed to?), to provide that guarantee. It just has to act as (following you example below) DMD does at the moment. But the spec has to reflect this, otherwise, a complying compiler can do anything.
 Your examples are flawed, declaring a variable with const is equivalent to

(or at least that
 is what most people expect, I don't really know if the spec says anything on
it).

My examples are not flawed, that is what the discussion was about in the beginning: rewriting the foreach range loop into the second example. Then bearophile protested.
 void main(){
     int a = 53;
     const int* pa = &a;
     auto pb = cast(int*) pa;
     assert(*pb == *pa);
     *pb = 3;
     assert(*pb == *pa);
 }

 This will always be true on every implementation.

No. Not as long as the behavior is undefined. All existing implementations, true. But all existing implementations use the same front-end so that is no argument at all. It is certainly not true for the general case of 'all [possible] implementations'. As long as the spec does not define what happens if you modify const, _the compiler can do anything!_. Does not mean that it will do anything specific, such as that what you'd like it to do, just that theoretically, it *could* format your hard drive, call your grandma, or steal your identity without breaking language imposed rules. You may want to have a look at the excellent series about the topic located at http://blog.regehr.org/archives/213 When I am talking about undefined behavior, I mean this. Cheers, - Timon
Jun 17 2011
parent reply Jesse Phillips <jessekphillips+D gmail.com> writes:
Timon Gehr Wrote:

 I will now do the same thing you did: Yes I know where it resides, I can
 disassemble the resulting executable and there is the address, plain as day!

Not possible, the executable can not contain an address because nothing is in memory to have an address.
 JK, I guess you did not realize there was a "...and mutating the memory" part
in
 my statement? =)

I did see that but you must first understand that casting away const is defined. Once this is done though the compiler can no longer identify when something that was once const is now being modified. The behavior of modifying an int* is also defined, and when you cast away const to get an int* you maybe modifying what use to be a const variables, but the compiler does not know that.
 at compile time. The compiler doesn't know where in memory the values are
stored

anything (and why do you think it is not allowed to?), to provide that guarantee. It just has to act as (following you example below) DMD does at the moment. But the spec has to reflect this, otherwise, a complying compiler can do anything.

No it can not provide that guarantee at no cost. It is allowed to do optimizations on const variables, I can not tell what optimizations the compiler could preform as I don't think there are any at this time, but why restrict future geniuses? Sorry shouldn't have brought up extra code, if you can come up with some code or design that can identify when a const value has changed, within the current guidelines of the spec the more power to you.
 My examples are not flawed, that is what the discussion was about in the
 beginning: rewriting the foreach range loop into the second example. Then
 bearophile protested.

This is an implementation detail of foreach that can be made to work with DMD. Though this does bring up a point you aren't trying to make. There are places where the compiler can statically identify you are modifying const and put in any code it wants, but it still can't define what happens when you modify const.
 
 void main(){
     int a = 53;
     const int* pa = &a;
     auto pb = cast(int*) pa;
     assert(*pb == *pa);
     *pb = 3;
     assert(*pb == *pa);
 }

 This will always be true on every implementation.

No. Not as long as the behavior is undefined.

Yes even if it is undefined.
 As long as the spec does not define what
 happens if you modify const, _the compiler can do anything!_. Does not mean
that
 it will do anything specific, such as that what you'd like it to do, just that
 theoretically, it *could* format your hard drive, call your grandma, or steal
your
 identity without breaking language imposed rules.

You are correct, that it could theoretically do all those things and still be valid to the spec. But it can't be done, not without breaking some other aspect of the spec.
 You may want to have a look at the excellent series about the topic located at
 http://blog.regehr.org/archives/213
 
 When I am talking about undefined behavior, I mean this.

Yes I've read his posts. And many of them the compiler knows what is happening or the hardware is even able to detect the behavior. But in the case of modifying const, you can't identify it to make it defined. I haven't been touching on the main reason it isn't defined, could be an immutable value. You are just asking for defining when it does point to a mutable value. It is like trying to define gravity as the force that keeps you on the earth. It does do a good job of that but only because of the properties of gravity.
Jun 17 2011
parent Timon Gehr <timon.gehr gmx.ch> writes:
Jesse Phillips wrote:
 Timon Gehr Wrote:

 I will now do the same thing you did: Yes I know where it resides, I can
 disassemble the resulting executable and there is the address, plain as day!

Not possible, the executable can not contain an address because nothing is in

Uh... You don't actually know the internals of executables, do you? Because if you do, please clarify your statement.
 JK, I guess you did not realize there was a "...and mutating the memory" part
in
 my statement? =)

I did see that but you must first understand that casting away const is defined.

was once
 const is now being modified.

I understand that.
 The behavior of modifying an int* is also defined, and when you cast away const

compiler does not > know that. Nope. As soon as you invoke undefined behavior, even the behavior of modifying an int* is not defined anymore. Encountering undefined behavior makes your whole program nonsensical. It does not matter if the compiler knows that. It could know if its developer wanted it to.
 at compile time. The compiler doesn't know where in memory the values are
stored

anything (and why do you think it is not allowed to?), to provide that guarantee. It just has to act as (following you example below) DMD does at the moment. But the spec has to reflect this, otherwise, a complying compiler can do anything.

No it can not provide that guarantee at no cost. It is allowed to do

could preform as I don't > think there are any at this time, but why restrict future geniuses? You cannot perform optimizations on const variables, because they don't give more guarantees than mutable variables. They only impose restrictions. It is as simple as that. If you want guarantees and optimizations, use immutable variables.
 Sorry shouldn't have brought up extra code, if you can come up with some code
or

guidelines of the > spec the more power to you. You can perform a runtime-check. But you don't need to to give the guarantee that casting away const from a reference to mutable memory will give you a valid reference through which that memory can be modified.
 My examples are not flawed, that is what the discussion was about in the
 beginning: rewriting the foreach range loop into the second example. Then
 bearophile protested.

This is an implementation detail of foreach that can be made to work with DMD.

I think it is not worth the special casing, but sure.
 Though this does bring up a point you aren't trying to make. There are places

code it wants, > but it still can't define what happens when you modify const. Sure, because the const could reference immutable memory.
 void main(){
     int a = 53;
     const int* pa = &a;
     auto pb = cast(int*) pa;
     assert(*pb == *pa);
     *pb = 3;
     assert(*pb == *pa);
 }

 This will always be true on every implementation.

No. Not as long as the behavior is undefined.

Yes even if it is undefined.

defined behavior <=> it must always behave that way on every implementation. undefined behavior <=> the behavior could be anything, but it might also be what you expected. There is nothing more to it.
 As long as the spec does not define what
 happens if you modify const, _the compiler can do anything!_. Does not mean
that
 it will do anything specific, such as that what you'd like it to do, just that
 theoretically, it *could* format your hard drive, call your grandma, or steal
your
 identity without breaking language imposed rules.

You are correct, that it could theoretically do all those things and still be

the spec. So it is theoretically possible, yet impossible? This fails to sound like a fair point. Which aspect would be broken?
 You may want to have a look at the excellent series about the topic located at
 http://blog.regehr.org/archives/213

 When I am talking about undefined behavior, I mean this.

Yes I've read his posts. And many of them the compiler knows what is happening

const, you
 can't identify it to make it defined.

You don't need to identify it to make the behavior defined just if it is actually pointing to mutable memory. The compiler does not know what is going on, it is simply ignoring it to generate faster code within the spec.
 I haven't been touching on the main reason it isn't defined, could be an

value. It is like
 trying to define gravity as the force that keeps you on the earth. It does do a

Your analogy has nothing to do with the issue. A better one. You have two classes of items. 1. Tea Cups. Dropping them to the floor will break them. 2. The magical flubber slime ball from outer space (TMFSBFOS). Dropping it to the floor will have undefined behavior. Now you have a set of equal boxes, each containing either a Tea Cup or the magical flubber slime ball from outer space. You have no way to know what it is. Now, a quick wrap-up of our points: Timon: Now what happens if you drop one of these boxes? Jesse: The behavior is undefined, because it could contain the magical flubber slime ball from outer space! Timon: And if there is a tea cup in there? Jesse: The behavior is undefined, because it could contain the magical flubber slime ball from outer space! Timon: Well, no it is a tea cup. What happens if you drop a tea cup? Jesse: It breaks. Timon: Yes, and if it is in a box? Jesse: The behavior is still undefined! Timon: No. The tea cup breaks. Jesse: Yes that is what happens, always happened and will always happen in all circumstances, yet the behavior is undefined. Also, white is black. Cheers, -Timon
Jun 17 2011
prev sibling next sibling parent David Nadlinger <see klickverbot.at> writes:
On 6/15/11 4:35 PM, Caligo wrote:
 The problem with trying to "fix" foreach is that it would create
 problems of its own.  The following code would not behave correctly:

    foreach(i; 0..10){
      if(i&  1)
        i += 1;
      writeln(i, " is even.");
    }

But in my opinion it should not be legal in the first place anyway, as we already have ┬╗continue┬ź to do this in a very clear way: --- foreach (i; 0 .. 10) { if (i & 1) continue; writeln(i, " is even."); } --- David
Jun 15 2011
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
Steven Schveighoffer wrote:
 On Wed, 15 Jun 2011 10:35:33 -0400, Caligo <iteronvexor gmail.com> wrote:

 You can create a temporary if you like:

   foreach(i; 0..10){
     int ii = i + 1;
     writeln(ii, " ");
   }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own.  The following code would not behave correctly:

   foreach(i; 0..10){
     if(i & 1)
       i += 1;
     writeln(i, " is even.");
   }

That code relies on undocumented behavior. It's likely to fail on some other D compiler. We can't worry about existing code that uses undocumented "features" of dmd. It's utilizing the knowledge of how D rewrites the foreach statement, not how foreach is documented to work. [snip.]

Actually that *is* exactly as it is documented to work. Any code exploiting it relies on documented behavior. See TDPL p. 74: foreach(<symbol>; <expression1> .. <expression2>) <statement> is rewritten to: { auto __n = <expression2>; auto symbol = true ? <expression1> : <expression2>; for(; <symbol> < __n; ++<symbol>) <statement> } This change would introduce an inconsistency with TDPL. Timon
Jun 15 2011
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
Steven Schveighoffer wrote:
 On Wed, 15 Jun 2011 13:09:40 -0400, Timon Gehr <timon.gehr gmx.ch> wrote:

 Steven Schveighoffer wrote:
 On Wed, 15 Jun 2011 10:35:33 -0400, Caligo <iteronvexor gmail.com>
 wrote:

 You can create a temporary if you like:

   foreach(i; 0..10){
     int ii = i + 1;
     writeln(ii, " ");
   }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own.  The following code would not behave correctly:

   foreach(i; 0..10){
     if(i & 1)
       i += 1;
     writeln(i, " is even.");
   }

That code relies on undocumented behavior. It's likely to fail on some other D compiler. We can't worry about existing code that uses undocumented "features" of dmd. It's utilizing the knowledge of how D rewrites the foreach statement, not how foreach is documented to work. [snip.]

Actually that *is* exactly as it is documented to work. Any code exploiting it relies on documented behavior. See TDPL p. 74: foreach(<symbol>; <expression1> .. <expression2>) <statement> is rewritten to: { auto __n = <expression2>; auto symbol = true ? <expression1> : <expression2>; for(; <symbol> < __n; ++<symbol>) <statement> } This change would introduce an inconsistency with TDPL.

I think it would be worth it. In any instance of foreach *except* this one, rebinding the value does not change the iteration. It doesn't fit with all other cases of foreach. -Steve

Well, it does make sense but we have to be careful here. If foreach range statements get less efficient underway, it is not worth it. I am perfectly fine with foreach(i;0..n){} just being syntactic sugar for for(int i=0;i<n;i++){}. Timon
Jun 15 2011
parent Timon Gehr <timon.gehr gmx.ch> writes:
Steven Schveighoffer wrote:
 I've seen several cases where the syntactic sugar does not work.

 two things:

 if the compiler detects you never change i, then it can eliminate the
 temporary variable.

It could. Does DMD do that? Let's see: import std.stdio; void main(){ foreach(_i;0..100){ writeln(i); } } void main(){ foreach(_i;0..100){ const i=_i; writeln(i); } } dmd test 1 mov -0x8(%rbp),%edi 2 mov -0x8(%rbp),%ecx 2 mov %rcx,%rdi dmd test -O Identical. Nice. Now for something more complex: import std.stdio; struct S{ int i; this(this){writeln("postblit");} void opUnary(string op:"++")(){++i;} } void main(){ foreach(_i;S(0)..S(2)){ //auto i=_i; alias _i i; } } dmd test -O With the alias i: With the auto i: postblit postblit => Not optimized away, even though it could remove the whole loop.
 We should favor the case which is most useful, not the one which is
 simplest to implement.  If you want surprising behavior, use a for loop.

Also if I want guaranteed efficient looping. :o)
 When I see foreach(i; xxx), I think i should take on every value contained
 in xxx, whether that be a range, array, or something else.  Being able to
 change the elements iterated during the loop is surprising to say the
 least.

 Also, think about this:

 foreach(i; 0..10)
 {
     i -= 2;
 }

 This is a loop that counts *down* to int.min from -1.  The idea that this
 should be *expected* behavior is just ludicrous to me.

 -Steve

Absolutely. Regarding consistency, it is pretty clear that it should be changed. Timon
Jun 15 2011
prev sibling next sibling parent Caligo <iteronvexor gmail.com> writes:
You can create a temporary if you like:

  foreach(i; 0..10){
    int ii = i + 1;
    writeln(ii, " ");
  }

Which outputs:
1 2 3 4 5 6 7 8 9 10


The problem with trying to "fix" foreach is that it would create
problems of its own.  The following code would not behave correctly:

  foreach(i; 0..10){
    if(i & 1)
      i += 1;
    writeln(i, " is even.");
  }

not to mention all the code that's already using foreach.
Jun 15 2011
prev sibling next sibling parent so <so so.so> writes:
 I think this is a bug.  Either i should be const during loop iteration,  
 or an additional temporary should be manufactured.

 It is always possible to get the above behavior using a for loop.   
 foreach should be sane.

I agree, you'd expect it to be passed by value. As mentioned reference must be stated explicitly. foreach(ref i...)
Jun 15 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 15 Jun 2011 10:35:33 -0400, Caligo <iteronvexor gmail.com> wrote:

 You can create a temporary if you like:

   foreach(i; 0..10){
     int ii = i + 1;
     writeln(ii, " ");
   }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own.  The following code would not behave correctly:

   foreach(i; 0..10){
     if(i & 1)
       i += 1;
     writeln(i, " is even.");
   }

That code relies on undocumented behavior. It's likely to fail on some other D compiler. We can't worry about existing code that uses undocumented "features" of dmd. It's utilizing the knowledge of how D rewrites the foreach statement, not how foreach is documented to work. Besides, I think this case is very uncommon. A better implementation for such a thing would be: foreach(i; 0..5) writeln(i * 2, " is even."); -Steve
Jun 15 2011
prev sibling next sibling parent Caligo <iteronvexor gmail.com> writes:
On Wed, Jun 15, 2011 at 9:44 AM, KennyTM~ <kennytm gmail.com> wrote:
 On Jun 15, 11 22:35, Caligo wrote:
 You can create a temporary if you like:

 =A0 foreach(i; 0..10){
 =A0 =A0 int ii =3D i + 1;
 =A0 =A0 writeln(ii, " ");
 =A0 }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own. =A0The following code would not behave correctly:

 =A0 foreach(i; 0..10){
 =A0 =A0 if(i& =A01)
 =A0 =A0 =A0 i +=3D 1;
 =A0 =A0 writeln(i, " is even.");
 =A0 }

 not to mention all the code that's already using foreach.

If the code rely on the 'implicit ref' behavior of 'i', the code is doing=

 wrong. The spec never mentioned what happened when the variable is modifi=

 in a ForeachRangeStatement. I believe it is more sane if modifying 'i' is
 disallowed unless 'ref' is used.

This should be a compile time error: foreach(i; 0..10){ i +=3D 1; write(i, " "); } This should be legal. foreach(ref i; 0..10){ i +=3D 1; write(i, " "); } Is that the general consensus here?
Jun 15 2011
prev sibling next sibling parent so <so so.so> writes:
On Wed, 15 Jun 2011 18:23:55 +0300, Caligo <iteronvexor gmail.com> wrote:

 This should be a compile time error:

   foreach(i; 0..10){
     i += 1;
     write(i, " ");
   }

 This should be legal.
   foreach(ref i; 0..10){
     i += 1;
     write(i, " ");
   }

 Is that the general consensus here?

They both should be legal, the first one is value, the second one is reference.
Jun 15 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 15 Jun 2011 11:23:55 -0400, Caligo <iteronvexor gmail.com> wrote:

 On Wed, Jun 15, 2011 at 9:44 AM, KennyTM~ <kennytm gmail.com> wrote:
 On Jun 15, 11 22:35, Caligo wrote:
 You can create a temporary if you like:

   foreach(i; 0..10){
     int ii = i + 1;
     writeln(ii, " ");
   }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own.  The following code would not behave correctly:

   foreach(i; 0..10){
     if(i&  1)
       i += 1;
     writeln(i, " is even.");
   }

 not to mention all the code that's already using foreach.

If the code rely on the 'implicit ref' behavior of 'i', the code is doing it wrong. The spec never mentioned what happened when the variable is modified in a ForeachRangeStatement. I believe it is more sane if modifying 'i' is disallowed unless 'ref' is used.

This should be a compile time error: foreach(i; 0..10){ i += 1; write(i, " "); }

No, it should be legal and write 1 2 3 4 5 6 7 8 9 10
 This should be legal.
   foreach(ref i; 0..10){
     i += 1;
     write(i, " ");
   }

If this is legal, it should write 1 3 5 7 9 I have no problem with it being legal. -Steve
Jun 15 2011
prev sibling next sibling parent Caligo <iteronvexor gmail.com> writes:
On Wed, Jun 15, 2011 at 10:34 AM, so <so so.so> wrote:
 On Wed, 15 Jun 2011 18:23:55 +0300, Caligo <iteronvexor gmail.com> wrote:

 This should be a compile time error:

 =A0foreach(i; 0..10){
 =A0 =A0i +=3D 1;
 =A0 =A0write(i, " ");
 =A0}

 This should be legal.
 =A0foreach(ref i; 0..10){
 =A0 =A0i +=3D 1;
 =A0 =A0write(i, " ");
 =A0}

 Is that the general consensus here?

They both should be legal, the first one is value, the second one is reference.

A reference to what exactly? Does this make any sense: for(ref int i; i < 10; ++i){ i +=3D 1; write(i, " "); } It doesn't even compile. 'foreach(i, 0..10){ }' is syntactic sugar for 'for(int i =3D 0; i < 10; ++i){ }', is it not?
Jun 15 2011
prev sibling next sibling parent so <so so.so> writes:
On Wed, 15 Jun 2011 19:08:29 +0300, Caligo <iteronvexor gmail.com> wrote:

 A reference to what exactly?

 Does this make any sense:
   for(ref int i; i < 10; ++i){
     i += 1;
     write(i, " ");
   }

 It doesn't even compile.

It is same as the other argument, the element, it should work the same way.
 'foreach(i, 0..10){ }' is syntactic sugar for 'for(int i = 0; i < 10;
 ++i){ }', is it not?

We don't know, and we don't need to know. It might be a wrapper over "while" too, since its a different entity it might have its own rules. And having element and the element index share same rules would be both consistent and the right thing to do.
Jun 15 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 15 Jun 2011 13:09:40 -0400, Timon Gehr <timon.gehr gmx.ch> wrote:

 Steven Schveighoffer wrote:
 On Wed, 15 Jun 2011 10:35:33 -0400, Caligo <iteronvexor gmail.com>  
 wrote:

 You can create a temporary if you like:

   foreach(i; 0..10){
     int ii = i + 1;
     writeln(ii, " ");
   }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own.  The following code would not behave correctly:

   foreach(i; 0..10){
     if(i & 1)
       i += 1;
     writeln(i, " is even.");
   }

That code relies on undocumented behavior. It's likely to fail on some other D compiler. We can't worry about existing code that uses undocumented "features" of dmd. It's utilizing the knowledge of how D rewrites the foreach statement, not how foreach is documented to work. [snip.]

Actually that *is* exactly as it is documented to work. Any code exploiting it relies on documented behavior. See TDPL p. 74: foreach(<symbol>; <expression1> .. <expression2>) <statement> is rewritten to: { auto __n = <expression2>; auto symbol = true ? <expression1> : <expression2>; for(; <symbol> < __n; ++<symbol>) <statement> } This change would introduce an inconsistency with TDPL.

I think it would be worth it. In any instance of foreach *except* this one, rebinding the value does not change the iteration. It doesn't fit with all other cases of foreach. -Steve
Jun 15 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 15 Jun 2011 13:56:04 -0400, Timon Gehr <timon.gehr gmx.ch> wrote:

 Steven Schveighoffer wrote:
 On Wed, 15 Jun 2011 13:09:40 -0400, Timon Gehr <timon.gehr gmx.ch>  
 wrote:

 Steven Schveighoffer wrote:
 On Wed, 15 Jun 2011 10:35:33 -0400, Caligo <iteronvexor gmail.com>
 wrote:

 You can create a temporary if you like:

   foreach(i; 0..10){
     int ii = i + 1;
     writeln(ii, " ");
   }

 Which outputs:
 1 2 3 4 5 6 7 8 9 10


 The problem with trying to "fix" foreach is that it would create
 problems of its own.  The following code would not behave correctly:

   foreach(i; 0..10){
     if(i & 1)
       i += 1;
     writeln(i, " is even.");
   }

That code relies on undocumented behavior. It's likely to fail on some other D compiler. We can't worry about existing code that uses undocumented "features" of dmd. It's utilizing the knowledge of how D rewrites the foreach statement, not how foreach is documented to work. [snip.]

Actually that *is* exactly as it is documented to work. Any code exploiting it relies on documented behavior. See TDPL p. 74: foreach(<symbol>; <expression1> .. <expression2>) <statement> is rewritten to: { auto __n = <expression2>; auto symbol = true ? <expression1> : <expression2>; for(; <symbol> < __n; ++<symbol>) <statement> } This change would introduce an inconsistency with TDPL.

I think it would be worth it. In any instance of foreach *except* this one, rebinding the value does not change the iteration. It doesn't fit with all other cases of foreach. -Steve

Well, it does make sense but we have to be careful here. If foreach range statements get less efficient underway, it is not worth it. I am perfectly fine with foreach(i;0..n){} just being syntactic sugar for for(int i=0;i<n;i++){}.

I've seen several cases where the syntactic sugar does not work. two things: if the compiler detects you never change i, then it can eliminate the temporary variable. We should favor the case which is most useful, not the one which is simplest to implement. If you want surprising behavior, use a for loop. When I see foreach(i; xxx), I think i should take on every value contained in xxx, whether that be a range, array, or something else. Being able to change the elements iterated during the loop is surprising to say the least. Also, think about this: foreach(i; 0..10) { i -= 2; } This is a loop that counts *down* to int.min from -1. The idea that this should be *expected* behavior is just ludicrous to me. -Steve
Jun 15 2011
prev sibling next sibling parent "Simen Kjaeraas" <simen.kjaras gmail.com> writes:
On Wed, 15 Jun 2011 22:40:24 +0200, bearophile <bearophileHUGS lycos.com>  
wrote:

 Don:

 The alternative would be to create __iterator1234 as const, and then
 cast away const in the ++ step. Ugly, but would eliminate the
 performance cost for structs.

Isn't it undefined in D to modify a const?

I believe this is only because the source may be immutable, and thus reside in ROM or otherwise be optimized away in places. Casting away const for a variable you have full control of, should not pose problems. -- Simen
Jun 16 2011
prev sibling next sibling parent Jesse Phillips <jessekphillips+d gmail.com> writes:
On Wed, 15 Jun 2011 16:40:24 -0400, bearophile wrote:

 Isn't it undefined in D to modify a const?

Yes, but unlike what every redditer is say, that doesn't mean the compiler can do whatever it wants. It means you can't define its behavior. Go ahead try. Throws an exception, nope, can't do that no information at run-time and const is already gone from a cast during compile-time. Modifies value, nope, might not be possible. Does nothing, nope, same as throwing exception.
Jun 17 2011
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-06-15 04:20, bearophile wrote:
 This post is about some characteristics of the D foreach that has nagged me
for some time.

 This is a little Python 2.6 program:


 for i in xrange(10):
      i += 1
      print i,


 Its output shows that you are allowed to modify the iteration variable
(contents of the iteration name), but the iteration goes on with no change:
 1 2 3 4 5 6 7 8 9 10


 Similar code in D using foreach shows a different story:

 import std.stdio;
 void main() {
      foreach (i; 0 .. 10) {
          i += 1;
          write(i, " ");
      }
 }

 The output:
 1 3 5 7 9


 In my opinion this is a bit bug-prone because in real code there is some risk
of modifying the iteration variable "i" by mistake. (Note: here I am not
talking about D for() loops. They are OK, their semantics is transparent
enough. foreach() loops are less transparent and they *look* higher level than
for() loops). I'd like the iteration variable to act as being a copy of the
true loop variable as in Python. If this is a bit bad for foreach performance,
then I'd like the compiler to forbid the mutation of the foreach iteration
variable inside the foreach body. Is this even possible?


 Currently you can't solve the problem adding a const(int) to the iteration
variable:

 import std.stdio;
 void main() {
      foreach (const(int) i; 0 .. 10) { // line 3
          write(i, " ");
      }
 }


 DMD gives:
 test.d(3): Error: variable test.main.i cannot modify const


 This is a related but different thing:
 http://d.puremagic.com/issues/show_bug.cgi?id=5255

 Bye,
 bearophile

I agree with this. The iteration variable should not be changeable by default and to get the current behavior one could add "ref". -- /Jacob Carlborg
Jun 17 2011
parent bearophile <bearophileHUGS lycos.com> writes:
Jacob Carlborg:

 I agree with this. The iteration variable should not be changeable by 
 default and to get the current behavior one could add "ref".

Thank you for all the answers. It seems most people agree with this idea (about 90%+ comments are positive). I think Andrei is not against this idea, if limited to normal numbers. I'd like to know if Walter is OK with this little change in D. Bye, bearophile
Jun 17 2011
prev sibling parent Jesse Phillips <jessekphillips+d gmail.com> writes:
On Sat, 18 Jun 2011 02:21:19 +0000, Timon Gehr wrote:

 You have two classes of items.
 
 1. Tea Cups. Dropping them to the floor will break them. 2. The magical
 flubber slime ball from outer space (TMFSBFOS). Dropping it to the floor
 will have undefined behavior.
 
 Now you have a set of equal boxes, each containing either a Tea Cup or
 the magical flubber slime ball from outer space. You have no way to know
 what it is.
 ...
 Cheers,
 -Timon

I really like the analogy you have set up. So let us define what happens when we drop the box. My position is that you can not define the resulting behavior. You claim that if the Tea Cup is in the box, the Tea Cup will break. I can agree that this will be what happens, but I can not define for you what dropping the box will do. But if the box were to be dropped with a Tea Cup was inside, and the Tea Cup turned into a whale, then it would still fulfill the spec, yet I can not provide for you a way that dropping a boxed Tea Cup could result in anything but a broken Tea Cup.
Jun 19 2011