• Saurabh Das (35/35) Sep 06 2018 Is this a bug with writeln?
• Simen =?UTF-8?B?S2rDpnLDpXM=?= (12/13) Sep 06 2018 Yup. What happens is writeln destructively iterates over b[i].
• Simen =?UTF-8?B?S2rDpnLDpXM=?= (3/15) Sep 06 2018 Issue: https://issues.dlang.org/show_bug.cgi?id=19229
• Jonathan M Davis (21/56) Sep 06 2018 It's not a bug in writeln. Any time that a range is copied, you must not...
• Steven Schveighoffer (8/67) Sep 06 2018 This is not necessarily true. It depends how the sub-ranges are returned...
• Jonathan M Davis (15/84) Sep 06 2018 That's still not a bug in formattedWrite. save only duplicates the
• Steven Schveighoffer (12/51) Sep 06 2018 That is the bug -- formattedWrite should save all the inner ranges
• Jonathan M Davis (42/102) Sep 06 2018 It's exactly what you're going to get in all cases if the ranges aren't
• Saurabh Das (4/4) Sep 09 2018 Thank you for explaining all this.
• Jonathan M Davis (17/20) Sep 10 2018 Ranges are fantastic, and the basic concept is solid, but a number of th...

Saurabh Das <saurabh.das gmail.com> writes:

Is this a bug with writeln?

void main()
{
     import std.stdio, std.range, std.algorithm;

     auto a1 = sort([1,3,5,4,2]);
     auto a2 = sort([9,8,9]);
     auto a3 = sort([5,4,5,4]);

     pragma(msg, typeof(a1));
     pragma(msg, typeof(a2));
     pragma(msg, typeof(a3));

     auto b = [a1, a2, a3];
     pragma(msg, typeof(b));

     writeln("b:");
     writeln(b);
     writeln(b);  // <-- this one prints incorrectly

     writeln("a:");
     writeln(a1);
     writeln(a2);
     writeln(a3);

}

Output
======

SortedRange!(int[], "a < b")
SortedRange!(int[], "a < b")
SortedRange!(int[], "a < b")
SortedRange!(int[], "a < b")[]
b:
[[1, 2, 3, 4, 5], [8, 9, 9], [4, 4, 5, 5]]
[[], [], []]
a:
[1, 2, 3, 4, 5]
[8, 9, 9]
[4, 4, 5, 5]

The issue goes away if I cast 'b' to const before writeln. I 
think it is a bug, but maybe I am missing something?

Simen =?UTF-8?B?S2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:

On Thursday, 6 September 2018 at 08:40:08 UTC, Saurabh Das wrote:
 Is this a bug with writeln?

Yup. What happens is writeln destructively iterates over b[i]. 
Since b[i] is a forward range, this shouldn't be done 
destructively. Instead, a copy should be made using b[i].save, 
somewhere deep in Phobos' bowels.

For now, here's a workaround:

writeln(b.dup);
writeln(b); // Look ma, it works twice!

.dup creates a copy of the array, and the destructive iteration 
happens on the copy instead.

--
   Simen

Simen =?UTF-8?B?S2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:

On Thursday, 6 September 2018 at 09:06:21 UTC, Simen Kjærås wrote:
 On Thursday, 6 September 2018 at 08:40:08 UTC, Saurabh Das 
 wrote:
 Is this a bug with writeln?

 Yup. What happens is writeln destructively iterates over b[i]. 
 Since b[i] is a forward range, this shouldn't be done 
 destructively. Instead, a copy should be made using b[i].save, 
 somewhere deep in Phobos' bowels.

 For now, here's a workaround:

 writeln(b.dup);
 writeln(b); // Look ma, it works twice!

 .dup creates a copy of the array, and the destructive iteration 
 happens on the copy instead.

Issue: https://issues.dlang.org/show_bug.cgi?id=19229
PR: https://github.com/dlang/phobos/pull/6695

Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:

On Thursday, September 6, 2018 2:40:08 AM MDT Saurabh Das via Digitalmars-d-
learn wrote:
 Is this a bug with writeln?

 void main()
 {
      import std.stdio, std.range, std.algorithm;

      auto a1 = sort([1,3,5,4,2]);
      auto a2 = sort([9,8,9]);
      auto a3 = sort([5,4,5,4]);

      pragma(msg, typeof(a1));
      pragma(msg, typeof(a2));
      pragma(msg, typeof(a3));

      auto b = [a1, a2, a3];
      pragma(msg, typeof(b));

      writeln("b:");
      writeln(b);
      writeln(b);  // <-- this one prints incorrectly

      writeln("a:");
      writeln(a1);
      writeln(a2);
      writeln(a3);

 }

 Output
 ======

 SortedRange!(int[], "a < b")
 SortedRange!(int[], "a < b")
 SortedRange!(int[], "a < b")
 SortedRange!(int[], "a < b")[]
 b:
 [[1, 2, 3, 4, 5], [8, 9, 9], [4, 4, 5, 5]]
 [[], [], []]
 a:
 [1, 2, 3, 4, 5]
 [8, 9, 9]
 [4, 4, 5, 5]

 The issue goes away if I cast 'b' to const before writeln. I
 think it is a bug, but maybe I am missing something?

It's not a bug in writeln. Any time that a range is copied, you must not do
_anything_ else with the original unless copying it is equivalent to calling
save on it, because the semantics of copying a range are unspecified. They
vary wildly depending on the range type (e.g. copying a dynamic array is
equivalent to calling save, but copying a class reference is not). When you
pass the range to writeln, you must assumed that it may have been consumed.
And since you have range of ranges, you must assume that the ranges that are
contained may have been consumed. If you want to pass them to writeln and
then do anything else with them, then you'll need to call save on every
range involved (which is a bit of a pain with a range of ranges, but it's
necessary all the same).

In many cases, you can get away with passing a range to a function or use it
with foreach and then continue to use it after that, but that's only because
copying those ranges is equivalent to calling save on them. It doesn't work
if any of the ranges involved aren't saved when they're copied, and it
doesn't work in generic code, because you have no clue whether the ranges
that are going to be used with that code are going to be saved when they're
copied.

- Jonathan M Davis

Steven Schveighoffer <schveiguy gmail.com> writes:

On 9/6/18 12:55 PM, Jonathan M Davis wrote:
 On Thursday, September 6, 2018 2:40:08 AM MDT Saurabh Das via Digitalmars-d-
 learn wrote:
 Is this a bug with writeln?

 void main()
 {
       import std.stdio, std.range, std.algorithm;

       auto a1 = sort([1,3,5,4,2]);
       auto a2 = sort([9,8,9]);
       auto a3 = sort([5,4,5,4]);

       pragma(msg, typeof(a1));
       pragma(msg, typeof(a2));
       pragma(msg, typeof(a3));

       auto b = [a1, a2, a3];
       pragma(msg, typeof(b));

       writeln("b:");
       writeln(b);
       writeln(b);  // <-- this one prints incorrectly

       writeln("a:");
       writeln(a1);
       writeln(a2);
       writeln(a3);

 }

 Output
 ======

 SortedRange!(int[], "a < b")
 SortedRange!(int[], "a < b")
 SortedRange!(int[], "a < b")
 SortedRange!(int[], "a < b")[]
 b:
 [[1, 2, 3, 4, 5], [8, 9, 9], [4, 4, 5, 5]]
 [[], [], []]
 a:
 [1, 2, 3, 4, 5]
 [8, 9, 9]
 [4, 4, 5, 5]

 The issue goes away if I cast 'b' to const before writeln. I
 think it is a bug, but maybe I am missing something?

 
 It's not a bug in writeln. Any time that a range is copied, you must not do
 _anything_ else with the original unless copying it is equivalent to calling
 save on it, because the semantics of copying a range are unspecified. They
 vary wildly depending on the range type (e.g. copying a dynamic array is
 equivalent to calling save, but copying a class reference is not). When you
 pass the range to writeln, you must assumed that it may have been consumed.
 And since you have range of ranges, you must assume that the ranges that are
 contained may have been consumed. If you want to pass them to writeln and
 then do anything else with them, then you'll need to call save on every
 range involved (which is a bit of a pain with a range of ranges, but it's
 necessary all the same).

This is not necessarily true. It depends how the sub-ranges are returned.

The bug is that formattedWrite takes ranges sometimes by ref, sometimes not.

formattedWrite should call save on a forward range whenever it makes a 
copy, and it doesn't.

Case in point, it doesn't matter if you call writeln(b.save), the same 
thing happens.

-Steve

Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:

On Thursday, September 6, 2018 12:21:24 PM MDT Steven Schveighoffer via 
Digitalmars-d-learn wrote:
 On 9/6/18 12:55 PM, Jonathan M Davis wrote:
 On Thursday, September 6, 2018 2:40:08 AM MDT Saurabh Das via
 Digitalmars-d->
 learn wrote:
 Is this a bug with writeln?

 void main()
 {

       import std.stdio, std.range, std.algorithm;

       auto a1 = sort([1,3,5,4,2]);
       auto a2 = sort([9,8,9]);
       auto a3 = sort([5,4,5,4]);

       pragma(msg, typeof(a1));
       pragma(msg, typeof(a2));
       pragma(msg, typeof(a3));

       auto b = [a1, a2, a3];
       pragma(msg, typeof(b));

       writeln("b:");
       writeln(b);
       writeln(b);  // <-- this one prints incorrectly

       writeln("a:");
       writeln(a1);
       writeln(a2);
       writeln(a3);

 }

 Output
 ======

 SortedRange!(int[], "a < b")
 SortedRange!(int[], "a < b")
 SortedRange!(int[], "a < b")
 SortedRange!(int[], "a < b")[]
 b:
 [[1, 2, 3, 4, 5], [8, 9, 9], [4, 4, 5, 5]]
 [[], [], []]
 a:
 [1, 2, 3, 4, 5]
 [8, 9, 9]
 [4, 4, 5, 5]

 The issue goes away if I cast 'b' to const before writeln. I
 think it is a bug, but maybe I am missing something?

 It's not a bug in writeln. Any time that a range is copied, you must not
 do _anything_ else with the original unless copying it is equivalent to
 calling save on it, because the semantics of copying a range are
 unspecified. They vary wildly depending on the range type (e.g. copying
 a dynamic array is equivalent to calling save, but copying a class
 reference is not). When you pass the range to writeln, you must assumed
 that it may have been consumed. And since you have range of ranges, you
 must assume that the ranges that are contained may have been consumed.
 If you want to pass them to writeln and then do anything else with
 them, then you'll need to call save on every range involved (which is a
 bit of a pain with a range of ranges, but it's necessary all the same).

 This is not necessarily true. It depends how the sub-ranges are returned.

 The bug is that formattedWrite takes ranges sometimes by ref, sometimes
 not.

 formattedWrite should call save on a forward range whenever it makes a
 copy, and it doesn't.

 Case in point, it doesn't matter if you call writeln(b.save), the same
 thing happens.

That's still not a bug in formattedWrite. save only duplicates the
outer-most range. And since writeln will ultimately iterate through the
inner ranges - which weren't saved - you end up with them being consumed.
When you're passing a range of ranges to a function, you need to recursively
save them if you don't want the inner ranges in the original range to be
consumed. Regardless of what formattedWrite does, it's a general issue with
any function that you pass a range of ranges. It comes right back to the
same issue of the semantics of copying ranges being unspecified and that you
therefore must always use save on any ranges involved if you want to then
use those ranges after having passed them to a function or copy them doing
anything else. It's that much more annoying when you're dealing with a range
of ranges rather than a range of something else, but the issue is the same.

- Jonathan M Davis

Steven Schveighoffer <schveiguy gmail.com> writes:

On 9/6/18 2:52 PM, Jonathan M Davis wrote:
 On Thursday, September 6, 2018 12:21:24 PM MDT Steven Schveighoffer via
 Digitalmars-d-learn wrote:
 On 9/6/18 12:55 PM, Jonathan M Davis wrote:
 It's not a bug in writeln. Any time that a range is copied, you must not
 do _anything_ else with the original unless copying it is equivalent to
 calling save on it, because the semantics of copying a range are
 unspecified. They vary wildly depending on the range type (e.g. copying
 a dynamic array is equivalent to calling save, but copying a class
 reference is not). When you pass the range to writeln, you must assumed
 that it may have been consumed. And since you have range of ranges, you
 must assume that the ranges that are contained may have been consumed.
 If you want to pass them to writeln and then do anything else with
 them, then you'll need to call save on every range involved (which is a
 bit of a pain with a range of ranges, but it's necessary all the same).

 This is not necessarily true. It depends how the sub-ranges are returned.

 The bug is that formattedWrite takes ranges sometimes by ref, sometimes
 not.

 formattedWrite should call save on a forward range whenever it makes a
 copy, and it doesn't.

 Case in point, it doesn't matter if you call writeln(b.save), the same
 thing happens.

 
 That's still not a bug in formattedWrite. save only duplicates the
 outer-most range. And since writeln will ultimately iterate through the
 inner ranges - which weren't saved - you end up with them being consumed.

That is the bug -- formattedWrite should save all the inner ranges 
(writeln calls formattedWrite, and lets it do all the work). To not do 
so leaves it open to problems such as consuming the sub ranges.

I can't imagine that anyone would expect or desire the current behavior.

Ironically, when that bug is fixed, you *don't* have to call save on the 
outer range!

 When you're passing a range of ranges to a function, you need to recursively
 save them if you don't want the inner ranges in the original range to be
 consumed. Regardless of what formattedWrite does, it's a general issue with
 any function that you pass a range of ranges. It comes right back to the
 same issue of the semantics of copying ranges being unspecified and that you
 therefore must always use save on any ranges involved if you want to then
 use those ranges after having passed them to a function or copy them doing
 anything else. It's that much more annoying when you're dealing with a range
 of ranges rather than a range of something else, but the issue is the same.
 

It's only a problem if the subranges are returned by reference. If they 
aren't, then no save is required (because they are already copies). The 
fix in this case is to make a copy if possible (using save as expected).

I think the save semantics have to be one of the worst designs in D.

-Steve

Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:

On Thursday, September 6, 2018 1:05:03 PM MDT Steven Schveighoffer via 
Digitalmars-d-learn wrote:
 On 9/6/18 2:52 PM, Jonathan M Davis wrote:
 On Thursday, September 6, 2018 12:21:24 PM MDT Steven Schveighoffer via

 Digitalmars-d-learn wrote:
 On 9/6/18 12:55 PM, Jonathan M Davis wrote:
 It's not a bug in writeln. Any time that a range is copied, you must
 not
 do _anything_ else with the original unless copying it is equivalent
 to
 calling save on it, because the semantics of copying a range are
 unspecified. They vary wildly depending on the range type (e.g.
 copying
 a dynamic array is equivalent to calling save, but copying a class
 reference is not). When you pass the range to writeln, you must
 assumed
 that it may have been consumed. And since you have range of ranges,
 you
 must assume that the ranges that are contained may have been consumed.
 If you want to pass them to writeln and then do anything else with
 them, then you'll need to call save on every range involved (which is
 a
 bit of a pain with a range of ranges, but it's necessary all the
 same).

 This is not necessarily true. It depends how the sub-ranges are
 returned.

 The bug is that formattedWrite takes ranges sometimes by ref, sometimes
 not.

 formattedWrite should call save on a forward range whenever it makes a
 copy, and it doesn't.

 Case in point, it doesn't matter if you call writeln(b.save), the same
 thing happens.

 That's still not a bug in formattedWrite. save only duplicates the
 outer-most range. And since writeln will ultimately iterate through the
 inner ranges - which weren't saved - you end up with them being
 consumed.

 That is the bug -- formattedWrite should save all the inner ranges
 (writeln calls formattedWrite, and lets it do all the work). To not do
 so leaves it open to problems such as consuming the sub ranges.

 I can't imagine that anyone would expect or desire the current behavior.

It's exactly what you're going to get in all cases if the ranges aren't
forward ranges, and it's what you have to do in general when passing ranges
of ranges to functions if you want to be able to continue to use any of the
ranges involved after passing them to the function. Changing formattedWrite
to work around it is only a workaround for this paricular case. It's still
a bug in general - though given that this would be one of the more common
cases, working around it in this particular case may be worth it. It's still
a workaround though and not something that can be relied on in with
range-based code in general - especially when most range-based code isn't
written to care about what the element types are and copies elements around
all the time.

 Ironically, when that bug is fixed, you *don't* have to call save on the
 outer range!

Except you do, because it's passed by value. If it's a dynamic array, then
you're fine, since copying saves, but in the general case, you still do.

 When you're passing a range of ranges to a function, you need to
 recursively save them if you don't want the inner ranges in the
 original range to be consumed. Regardless of what formattedWrite does,
 it's a general issue with any function that you pass a range of ranges.
 It comes right back to the same issue of the semantics of copying
 ranges being unspecified and that you therefore must always use save on
 any ranges involved if you want to then use those ranges after having
 passed them to a function or copy them doing anything else. It's that
 much more annoying when you're dealing with a range of ranges rather
 than a range of something else, but the issue is the same.

 It's only a problem if the subranges are returned by reference. If they
 aren't, then no save is required (because they are already copies). The
 fix in this case is to make a copy if possible (using save as expected).

 I think the save semantics have to be one of the worst designs in D.

On that we can definitely agree. I'm strongly of the opinion that it should
have been required that forward ranges be dynamic arrays or structs (no
classes allowed) and that it be required that they have a postblit / copy
constructor if the default copy wasn't equivalent to save. If you wanted a
class that was a forward range, you would then have to wrap it in a struct
with the appropriate postblit / copy constructor. That way, copying a
forward range would _always_ be saving it.

The harder question is what to then do with basic input ranges. Having them
share code with forward ranges is often useful but also frequently a
disaster, and to really be correct, they would need to either be full-on
reference types or always passed around by reference. Allowing partial
reference types is a total disaster when you're allowed to copy the range.
Requiring that they be classes would fix the problem nicely except that it
would then require heap allocation for all basic input ranges.

Basic input ranges and forward ranges are the same when all you're doing is
iterating over them, but as soon as you pass them around - even to a foreach
loop (since a foreach loop copie the range) - it gets hairy fast.
Semantically, basic input ranges really should be reference types, whereas
forward ranges really should be value types (or at least their outer layer
needs to act like a value type - deep copying would not be good, whereas the
behavior that dynamic arrays have would be fine).

So, I don't know exactly what I'd do if we could do this from scratch, but I
am quite sure that I would not have save be a thing. That really needs to
just be what happens when copying normally. Unfortunately, we can't exactly
fix that now. Ranges make up what is arguably our greatest set of idioms,
but they're also where we made some of our greatest mistakes. *sigh*

- Jonathan M Davis

Saurabh Das <saurabh.das gmail.com> writes:

Thank you for explaining all this.

It is frustrating because the behaviour is very counterintuitive.

I will use a workaround for now.

Saurabh

Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:

On Sunday, September 9, 2018 8:30:12 AM MDT Saurabh Das via Digitalmars-d-
learn wrote:
 Thank you for explaining all this.

 It is frustrating because the behaviour is very counterintuitive.

 I will use a workaround for now.

Ranges are fantastic, and the basic concept is solid, but a number of the
implementation details really weren't worked out well enough in the
beginning, which means that there's a lot of stuff that works reasonably
well with ranges but falls apart on some level in certain circumstances -
and what happens when ranges get copied is a big part of that.

On some level, it's what we get for being the first language to really
implement ranges as part of its standard library. As I understand it, Andrei
didn't create the concept, but AFAIK, it wasn't implemented on any real
scale by anyone prior to them being added to Phobos. And when you're the
first to implement something, you often screw it up on some level. With many
things, D was able to learn from C++ and improve, but this is an area where
we got to make new mistakes. The result is something that's extremely useful
and largely works but which has some problematic corner cases that really
shouldn't be there.

- Jonathan M Davis