www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Is run.d going to be expand for runtime and the phobos library?

reply 12345swordy <alexanderheistermann gmail.com> writes:
Quite curious that I never see any attempt on it what so ever.
Jun 11 2020
next sibling parent reply Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Friday, 12 June 2020 at 00:24:41 UTC, 12345swordy wrote:

 Quite curious that I never see any attempt on it what so ever.


You leave me guessing what run.d is, and so I have no idea what 
you are curious about.

— Bastiaan.
Jun 12 2020
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Friday, 12 June 2020 at 07:38:22 UTC, Bastiaan Veelo wrote:

 On Friday, 12 June 2020 at 00:24:41 UTC, 12345swordy wrote:

 Quite curious that I never see any attempt on it what so ever.


 You leave me guessing what run.d is, and so I have no idea what 
 you are curious about.

 — Bastiaan.


(This is a message from a day ago that I forgot to send earlier.)

TL;DR
See the Readme file here: 
https://github.com/dlang/dmd/tree/v2.092.1/test and run.d itself: 
https://github.com/dlang/dmd/blob/v2.092.1/test/run.d.
Also you can check: 
https://github.com/dlang/dmd/blob/v2.092.1/src/build.d

---

The basic idea is that we're replacing the ugly makefile mess 
with a build and test system written in pure D. The advantages 
going for a pure D solution are numerous:
- Developers and package maintainers need less external 
dependencies to build and test the code
- There are several Make(1) implementations (e.g. GNUmake, 
BSDmake, DMmake [1]) each with different feature sets and 
limitations. This was especially a problem on Windows, as DMmake 
has much more limited feature set then GNUmake, which prevented 
us from having a single cross-platform Makefile and instead we 
had to maintain posix.mak, win32.mak and win64.mak
- AFAIK, installing a GNUmake on Windows involves installing a 
whole Posix emulation/compatibility environment like cygwin, msys 
or msys2, which brings it's own rats nest of problems
- Even if Make is not a problem, it is still dependent on Posix 
compatible shell, so that's one more variable in the equation
- By implementing the full build system in D we have much more 
flexibility and it's easier to troubleshoot problems. Also we can 
implement features which are unlikely to ever become part of 
make, but would be useful for us.
- For example, it's we can tune the scheduling of the test runner 
relatively easily, while with Make we he no control over that 
(it's just doing topological sorting, without any semantic 
understanding of what each target represents).
- And quite a few other benefits...


[1]: https://digitalmars.com/ctg/make.html
Jun 13 2020
parent reply kinke <noone nowhere.com> writes:
On Saturday, 13 June 2020 at 08:58:56 UTC, Petar Kirov 
[ZombineDev] wrote:

 - AFAIK, installing a GNUmake on Windows involves installing a 
 whole Posix emulation/compatibility environment like cygwin, 
 msys or msys2, which brings it's own rats nest of problems
 - Even if Make is not a problem, it is still dependent on Posix 
 compatible shell, so that's one more variable in the equation


Side note: LDC has been using GNUmake on Windows for years, incl. 
extending some upstream Makefiles to work with and test Windows 
targets too.
While GNUmake itself is surprisingly portable (source comes with 
a Visual Studio project working out of the box, resulting in a 
little 330 KB Win64 stand-alone executable), the tests require 
bash and the usual little GNU tools.
Fortunately, a normal git installation on Windows provides the 
expected GNU environment, incl. bash.
Jun 13 2020
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/13/2020 6:44 AM, kinke wrote:

 Side note: LDC has been using GNUmake on Windows for years, incl. extending
some 
 upstream Makefiles to work with and test Windows targets too.
 While GNUmake itself is surprisingly portable (source comes with a Visual
Studio 
 project working out of the box, resulting in a little 330 KB Win64 stand-alone 
 executable),


DM make is 52K :-)
Jun 13 2020
parent reply Avrina <avrina12309412342 gmail.com> writes:
On Sunday, 14 June 2020 at 04:26:00 UTC, Walter Bright wrote:

 On 6/13/2020 6:44 AM, kinke wrote:

 Side note: LDC has been using GNUmake on Windows for years, 
 incl. extending some upstream Makefiles to work with and test 
 Windows targets too.
 While GNUmake itself is surprisingly portable (source comes 
 with a Visual Studio project working out of the box, resulting 
 in a little 330 KB Win64 stand-alone executable),


 DM make is 52K :-)


Size doesn't matter if it doesn't work.
Jun 14 2020
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/14/2020 9:18 AM, Avrina wrote:

 Size doesn't matter if it doesn't work.


I use it all the time, it works fine. It does lack the features that make 
makefiles incomprehensible, though.
Jun 20 2020
next sibling parent Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Saturday, 20 June 2020 at 20:13:26 UTC, Walter Bright wrote:

 On 6/14/2020 9:18 AM, Avrina wrote:

 Size doesn't matter if it doesn't work.


 I use it all the time, it works fine.


This one is preventing me from using optlink: 
https://issues.dlang.org/show_bug.cgi?id=15213

—Bastiaan
Jun 20 2020
prev sibling parent Avrina <avrina12309412342 gmail.com> writes:
On Saturday, 20 June 2020 at 20:13:26 UTC, Walter Bright wrote:

 On 6/14/2020 9:18 AM, Avrina wrote:

 Size doesn't matter if it doesn't work.


 I use it all the time, it works fine. It does lack the features 
 that make makefiles incomprehensible, though.


I use it none of the time, and loath the fact it is named "make" 
which makes it's way into PATH. It's why it was replaced in DMD, 
it doesn't do what it needs to.

You mean comprehensible DM make files like this?

$G/nteh.obj : $C\rtlsym.h $C\nteh.c
	$(CC) -c -o$  $(MFLAGS) $C\nteh

$G/os.obj : $C\os.c
	$(CC) -c -o$  $(MFLAGS) $C\os

$G/out.obj : $C\out.c
	$(CC) -c -o$  $(MFLAGS) $C\out

$G/outbuf.obj : $C\outbuf.h $C\outbuf.c
	$(CC) -c -o$  $(MFLAGS) $C\outbuf

$G/pdata.obj : $C\pdata.c
	$(CC) -c -o$  $(MFLAGS) $C\pdata

$G/ph2.obj : $C\ph2.c
	$(CC) -c -o$  $(MFLAGS) $C\ph2

Yes so clear.
Jun 21 2020
prev sibling parent reply MoonlightSentinel <moonlightsentinel disroot.org> writes:
On Friday, 12 June 2020 at 00:24:41 UTC, 12345swordy wrote:

 Quite curious that I never see any attempt on it what so ever.


I assume you are talking about DMD's test runner 
(https://github.com/dlang/dmd/blob/master/test/run.d).


There are no concrete plan's AFAICT but I generally like the idea 
to replace make for the other repos as well.
Jun 12 2020
parent reply 12345swordy <alexanderheistermann gmail.com> writes:
On Friday, 12 June 2020 at 08:27:07 UTC, MoonlightSentinel wrote:

 On Friday, 12 June 2020 at 00:24:41 UTC, 12345swordy wrote:

 Quite curious that I never see any attempt on it what so ever.


 I assume you are talking about DMD's test runner 
 (https://github.com/dlang/dmd/blob/master/test/run.d).


 There are no concrete plan's AFAICT but I generally like the 
 idea to replace make for the other repos as well.


I'm Sorry, I was referring to this:
https://github.com/dlang/dmd/blob/master/src/build.d
Jun 12 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/12/20 9:42 AM, 12345swordy wrote:

 On Friday, 12 June 2020 at 08:27:07 UTC, MoonlightSentinel wrote:

 On Friday, 12 June 2020 at 00:24:41 UTC, 12345swordy wrote:

 Quite curious that I never see any attempt on it what so ever.


 I assume you are talking about DMD's test runner 
 (https://github.com/dlang/dmd/blob/master/test/run.d).


 There are no concrete plan's AFAICT but I generally like the idea to 
 replace make for the other repos as well.


 
 I'm Sorry, I was referring to this:
 https://github.com/dlang/dmd/blob/master/src/build.d


That should be killed with fire. I have seldom disliked a program this much.
Jun 12 2020
next sibling parent reply Seb <seb wilzba.ch> writes:
On Saturday, 13 June 2020 at 03:33:14 UTC, Andrei Alexandrescu 
wrote:

 On 6/12/20 9:42 AM, 12345swordy wrote:

 On Friday, 12 June 2020 at 08:27:07 UTC, MoonlightSentinel 
 wrote:

 On Friday, 12 June 2020 at 00:24:41 UTC, 12345swordy wrote:

 Quite curious that I never see any attempt on it what so 
 ever.


 I assume you are talking about DMD's test runner 
 (https://github.com/dlang/dmd/blob/master/test/run.d).


 There are no concrete plan's AFAICT but I generally like the 
 idea to replace make for the other repos as well.


 
 I'm Sorry, I was referring to this:
 https://github.com/dlang/dmd/blob/master/src/build.d


 That should be killed with fire. I have seldom disliked a 
 program this much.


Huh. The Andrei I remember approved the migration away from the 
DigitalMars Makefiles multiple times.

On the contrary, there are plans to expand build.d for 
druntime+phobos and the only reason it didn't happen is because I 
myself consider Phobos as low priority to maintain as it's 
basically frozen/dead code.

Anyhow, PRs for build.d at druntime/phobos would be merged if 
someone is interested in this.
Jun 13 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/13/20 4:30 AM, Seb wrote:

 I myself consider Phobos as low priority to maintain as it's basically 
 frozen/dead code.


I was wondering what would be the drawbacks of defining an ultra-simple 
convention for versions of the standard library - with yearly 
granularity. Not being an expert in versioning I've always been coy to 
mention it, but how about trying it instead of the current stalemate. 
After all C++ does it (both with 3-year language versions and with 
things like std::tr1, std::tr2 etc) and it has a more dangerous 
modularity mechanism.

D's modularity is, or should be, rock solid. So then we could simply 
define "vYEAR" as versions of standard library modules. So:

// import the backward-compatible lib
import std.algorithm;
// import this year's algorithm with breaking changes
import std.v2020.algorithm;
// live dangerously, import work-in-progress
import std.v2020.algorithm;

Of course no need to define yearly versions for every year and every 
package/module. Just those that are worth adding backwards-breaking 
improvements.

A module should not import both std.xyz and std.v2020.xyz. Or the 
behavior is dependent on xyz and defined clearly.

I wonder how well this would work.
Jun 13 2020
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/13/20 3:42 PM, Andrei Alexandrescu wrote:

 On 6/13/20 4:30 AM, Seb wrote:

 I myself consider Phobos as low priority to maintain as it's basically 
 frozen/dead code.


 
 I was wondering what would be the drawbacks of defining an ultra-simple 
 convention for versions of the standard library - with yearly 
 granularity. Not being an expert in versioning I've always been coy to 
 mention it, but how about trying it instead of the current stalemate. 
 After all C++ does it (both with 3-year language versions and with 
 things like std::tr1, std::tr2 etc) and it has a more dangerous 
 modularity mechanism.
 
 D's modularity is, or should be, rock solid. So then we could simply 
 define "vYEAR" as versions of standard library modules. So:
 
 // import the backward-compatible lib
 import std.algorithm;
 // import this year's algorithm with breaking changes
 import std.v2020.algorithm;
 // live dangerously, import work-in-progress
 import std.v2020.algorithm;


Eh, meant v2021 here.
Jun 13 2020
parent reply jmh530 <john.michael.hall gmail.com> writes:
On Saturday, 13 June 2020 at 19:52:16 UTC, Andrei Alexandrescu 
wrote:

 [snip]

 D's modularity is, or should be, rock solid. So then we could 
 simply define "vYEAR" as versions of standard library modules. 
 So:
 
 // import the backward-compatible lib
 import std.algorithm;
 // import this year's algorithm with breaking changes
 import std.v2020.algorithm;
 // live dangerously, import work-in-progress
 import std.v2020.algorithm;


 Eh, meant v2021 here.


Something like below allows the user to specify what they want 
for all of phobos at the command or to import the specific one 
they want.

module std.algorithm;

version(std_latest) {
     version = std_2_92;
}
version(std_stable) {
     version = std_2_89;
}
version(std_2019) {
     version = std_2_89;
}
version(std_2_92) {
     public import 2_92.std.algorithm;
}
version(std_2_91) {
     public import 2_91.std.algorithm;
}
version(std_2_90) {
     public import 2_90.std.algorithm;
}
version(std_2_89) {
     public import 2_89.std.algorithm;
}
...

One problem is that if you allow people to make bug fixes to old 
phobos versions, then there is no way for the subsequent releases 
to get the updates as if you could with a git branch. You would 
have to do them all manually.

I also saw a suggestion about C++'s epoch proposal on another 
thread that was interesting.
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p1881r0.html
Jun 13 2020
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/13/20 4:37 PM, jmh530 wrote:

 One problem is that if you allow people to make bug fixes to old phobos 
 versions, then there is no way for the subsequent releases to get the 
 updates as if you could with a git branch. You would have to do them all 
 manually.


It's not that bad. Typical newer iterations of phobos versions would 
simply list as public the unchanged symbols, so they are "deltas". Fixes 
to unchanged symbols would just propagate. Indeed fixes to changed 
symbols would need to be propagated manually, but even that may be 
mitigated by having the newer versions call the older versions on occasion.

More importantly versioning allows us to experiment with things such as 
"let's use assert and Expected instead of throwing throughout". These 
are definitely breaking APIs, but can be implemented easily in such a 
way that both the backward-compatible version and the new version share 
the same codebase. (You can wrap noexcept code in code that throws, etc.)


 I also saw a suggestion about C++'s epoch proposal on another thread 
 that was interesting.
 http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p1881r0.html


Thanks!
Jun 13 2020
prev sibling parent reply Jesse Phillips <Jesse.K.Phillips+D gmail.com> writes:
On Saturday, 13 June 2020 at 20:37:59 UTC, jmh530 wrote:

 One problem is that if you allow people to make bug fixes to 
 old phobos versions, then there is no way for the subsequent 
 releases to get the updates as if you could with a git branch. 
 You would have to do them all manually.


Git will allow you to create a patch from one file and apply it 
to another. Still manual.


Working in std.experimental I found the private symbols had to 
work with. That is likely not as bad with this since the idea 
would be to recreate the phobos ecosystem around the changes. To 
that end it may be best to take a module and it's entirety into 
the new version and forwarding only happens across modules.
Jun 18 2020
parent reply jmh530 <john.michael.hall gmail.com> writes:
On Thursday, 18 June 2020 at 14:16:13 UTC, Jesse Phillips wrote:

 On Saturday, 13 June 2020 at 20:37:59 UTC, jmh530 wrote:

 One problem is that if you allow people to make bug fixes to 
 old phobos versions, then there is no way for the subsequent 
 releases to get the updates as if you could with a git branch. 
 You would have to do them all manually.


 Git will allow you to create a patch from one file and apply it 
 to another. Still manual.

 [snip]


I think Andrei had made a good point about propagating up 
changes. This would imply keeping the structure that I had 
previously for std.algorithm overall, but modify the N_NN module 
files. For instance, if one were starting from 2_89, then that 
would be unchanged, while 2_90 would be

module 2_90.std.algorithm;
public import 2_89.std.algorithm: funA, funC, funD...;
void funB() {}

and so on up. For an even more fine-grained basis, you could do

module 2_90_0.std.algorithm;
public import 2_89.std.algorithm: funA, funC, funD...;
void funB() {}

module 2_90_1.std.algorithm;
public import 2_90_0.std.algorithm: funA, funC, funD...;
void funB() {}

module 2_90.std.algorithm;
public import 2_90_1.std.algorithm;

So this would work pretty well in terms of minimizing the amount 
of code re-use going on.

(there is probably scope for a mixin that can do those imports, 
or a new language feature "import module: -f" that imports 
everything except f)

With respect to your suggestion, suppose you release some patch 
for funB in 2.90.1 late, i.e. when the current version is 2.95 or 
something. If funB has not been changed over this period, then it 
propagates up without issue (note that the import of 2_89 would 
not be over the patch, 2_89 would be structured like 2_90). 
However, if funB has changed, then future updates will not 
benefit from it. You would have to manually update those as well. 
Your suggestion would be useful in this case, I think.
Jun 18 2020
next sibling parent reply Jesse Phillips <Jesse.K.Phillips+D gmail.com> writes:
On Thursday, 18 June 2020 at 15:55:06 UTC, jmh530 wrote:

 not benefit from it. You would have to manually update those as 
 well. Your suggestion would be useful in this case, I think.


I think my concern is mostly moot. I think managing the updates 
will be a challenge in all approaches. But I also think this is a 
good idea overall.
Jun 18 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/18/20 10:08 PM, Jesse Phillips wrote:

 On Thursday, 18 June 2020 at 15:55:06 UTC, jmh530 wrote:

 not benefit from it. You would have to manually update those as well. 
 Your suggestion would be useful in this case, I think.


 
 I think my concern is mostly moot. I think managing the updates will be 
 a challenge in all approaches. But I also think this is a good idea 
 overall.


One good goal for std.v2020 would be to forego autodecoding throughout. 
The endeavor will no doubt present challenges, which in turn will prompt 
clever solutions and a good accumulation of experience.
Jun 19 2020
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:
[...]

 One good goal for std.v2020 would be to forego autodecoding
 throughout.


[...]

Another could be to fix up the range API -- i.e, reconsider the ugliness
that is .save, now that D has copy ctors.


T

-- 
I'm still trying to find a pun for "punishment"...
Jun 19 2020
next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:

 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu 
 via Digitalmars-d wrote: [...]

 One good goal for std.v2020 would be to forego autodecoding 
 throughout.


 [...]

 Another could be to fix up the range API -- i.e, reconsider the 
 ugliness that is .save, now that D has copy ctors.


How do you see that? Fundamentally, what have copy ctors changed 
in this regard?
Jun 20 2020
next sibling parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Saturday, 20 June 2020 at 09:49:38 UTC, Stanislav Blinov wrote:

 On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:

 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu 
 via Digitalmars-d wrote: [...]

 One good goal for std.v2020 would be to forego autodecoding 
 throughout.


 [...]

 Another could be to fix up the range API -- i.e, reconsider 
 the ugliness that is .save, now that D has copy ctors.


 How do you see that? Fundamentally, what have copy ctors 
 changed in this regard?


I suppose that postblit has had various implementation and 
language design issues (which copy constructors address), which 
prevented them from being a reliable alternative to save(). This 
and probably also class support.
Jun 20 2020
next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 20 June 2020 at 10:03:06 UTC, Petar Kirov 
[ZombineDev] wrote:

 On Saturday, 20 June 2020 at 09:49:38 UTC, Stanislav Blinov 
 wrote:

 On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:





 Another could be to fix up the range API -- i.e, reconsider 
 the ugliness that is .save, now that D has copy ctors.


 How do you see that? Fundamentally, what have copy ctors 
 changed in this regard?


 I suppose that postblit has had various implementation and 
 language design issues (which copy constructors address), which 
 prevented them from being a reliable alternative to save(). 
 This and probably also class support.


All ranges are supposed to be copyable, so their copy-ability is 
not sufficient to distinguish input and forward ranges. Whether 
it's postblit or copy ctor makes no difference here, unless I'm 
missing something.
Jun 20 2020
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Saturday, 20 June 2020 at 10:52:37 UTC, Stanislav Blinov wrote:

 On Saturday, 20 June 2020 at 10:03:06 UTC, Petar Kirov 
 [ZombineDev] wrote:

 On Saturday, 20 June 2020 at 09:49:38 UTC, Stanislav Blinov 
 wrote:

 On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:





 Another could be to fix up the range API -- i.e, reconsider 
 the ugliness that is .save, now that D has copy ctors.


 How do you see that? Fundamentally, what have copy ctors 
 changed in this regard?


 I suppose that postblit has had various implementation and 
 language design issues (which copy constructors address), 
 which prevented them from being a reliable alternative to 
 save(). This and probably also class support.


 All ranges are supposed to be copyable, so their copy-ability 
 is not sufficient to distinguish input and forward ranges. 
 Whether it's postblit or copy ctor makes no difference here, 
 unless I'm missing something.


As far as I know, back when the ranges API was worked on postblit 
ctors didn't work reliably, so .save() was the only option. If it 
had worked, we could require that non-forward ranges are 
non-copyable.
Jun 20 2020
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 20 June 2020 at 11:20:13 UTC, Petar Kirov 
[ZombineDev] wrote:


 As far as I know, back when the ranges API was worked on 
 postblit ctors didn't work reliably, so .save() was the only 
 option.


Funny that, at the moment it's the copy ctors that aren't working 
reliably (e.g. they're not working at all with arrays). :)


 If it had worked, we could require that non-forward ranges are 
 non-copyable.


I can imagine this would be quite some work to adapt all of 
Phobos to *that*. I mean, things like

auto rem = makeSomeInputRange.find!pred;
auto flt = makeSomeInputRange.filter!pred;

Those would not compile. They could be made to compile by 
`move`ing the argument for the return. But then you still won't 
be able to pass those results around, unless via a refRange or 
`move`.
Jun 20 2020
next sibling parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Saturday, 20 June 2020 at 12:30:43 UTC, Stanislav Blinov wrote:

 On Saturday, 20 June 2020 at 11:20:13 UTC, Petar Kirov 
 [ZombineDev] wrote:


 As far as I know, back when the ranges API was worked on 
 postblit ctors didn't work reliably, so .save() was the only 
 option.


 Funny that, at the moment it's the copy ctors that aren't 
 working reliably (e.g. they're not working at all with arrays). 
 :)


You mean that array.dup doesn't work if the element type uses 
copy constructors? I haven't checked so I guess it's a problem 
indeed, though this shouldn't affect the ranges themselves (i.e. 
array slices).


 If it had worked, we could require that non-forward ranges are 
 non-copyable.


 I can imagine this would be quite some work to adapt all of 
 Phobos to *that*. I mean, things like

 auto rem = makeSomeInputRange.find!pred;
 auto flt = makeSomeInputRange.filter!pred;

 Those would not compile. They could be made to compile by 
 `move`ing the argument for the return. But then you still won't 
 be able to pass those results around, unless via a refRange or 
 `move`.


Yeah, I know. Though input-only range are odd in general. I think 
they can only work well either if the code moves them around, or 
if they use reference semantics (classes or refRange).
Jun 20 2020
next sibling parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 20 June 2020 at 13:01:33 UTC, Petar Kirov 
[ZombineDev] wrote:


 You mean that array.dup doesn't work if the element type uses 
 copy constructors? I haven't checked so I guess it's a problem 
 indeed, though this shouldn't affect the ranges themselves 
 (i.e. array slices).


Yes, Razvan Nitu is working on a fix for .dup already. But 
there's lots more. The ~= also doesn't call copy ctor. Nor does 
copying of slices and static arrays, i.e.

S[10] arr;
auto brr = arr; // no copy ctors called
arr[0 .. 2] = arr[2 .. 4]; // no copy ctors called

While .dup/.idup are a druntime-only fix, the rest requires fixes 
in the compiler.

It may be of small concern to ranges themselves (barring 
ranges-of-ranges), but a concern nonetheless.


 Yeah, I know. Though input-only range are odd in general. I 
 think they can only work well either if the code moves them 
 around, or if they use reference semantics (classes or 
 refRange).


But if they *do* use reference semantics, they need be copyable 
(see e.g. std.stdio.ByLineImpl). Requiring them to become classes 
I'd say would be too harsh. That said, I agree that most of them 
would be a consume-and-throw-away (i.e. temporaries). But there's 
always an odd case where you'd want to consume them in staggered 
steps.
Jun 20 2020
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/20 9:01 AM, Petar Kirov [ZombineDev] wrote:

 On Saturday, 20 June 2020 at 12:30:43 UTC, Stanislav Blinov wrote:

 On Saturday, 20 June 2020 at 11:20:13 UTC, Petar Kirov [ZombineDev] 
 wrote:


 As far as I know, back when the ranges API was worked on postblit 
 ctors didn't work reliably, so .save() was the only option.


 Funny that, at the moment it's the copy ctors that aren't working 
 reliably (e.g. they're not working at all with arrays). :)


 
 You mean that array.dup doesn't work if the element type uses copy 
 constructors? I haven't checked so I guess it's a problem indeed, though 
 this shouldn't affect the ranges themselves (i.e. array slices).


I thought this was fixed a while ago.
Jun 20 2020
parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 20 June 2020 at 14:54:49 UTC, Andrei Alexandrescu 
wrote:

 On 6/20/20 9:01 AM, Petar Kirov [ZombineDev] wrote:



 You mean that array.dup doesn't work if the element type uses 
 copy constructors? I haven't checked so I guess it's a problem 
 indeed, though this shouldn't affect the ranges themselves 
 (i.e. array slices).


 I thought this was fixed a while ago.


Huh?

https://github.com/dlang/druntime/pull/3139#discussion_r441924024

:)
Jun 20 2020
prev sibling next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Saturday, 20 June 2020 at 12:30:43 UTC, Stanislav Blinov wrote:

 If it had worked, we could require that non-forward ranges are 
 non-copyable.


 I can imagine this would be quite some work to adapt all of 
 Phobos to *that*. I mean, things like

 auto rem = makeSomeInputRange.find!pred;
 auto flt = makeSomeInputRange.filter!pred;

 Those would not compile. They could be made to compile by 
 `move`ing the argument for the return. But then you still won't 
 be able to pass those results around, unless via a refRange or 
 `move`.


https://github.com/WalterBright/DIPs/blob/13NNN-WGB.md/DIPs/13NNN-WGB.md
Jun 20 2020
parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 20 June 2020 at 13:09:54 UTC, Paul Backus wrote:


 https://github.com/WalterBright/DIPs/blob/13NNN-WGB.md/DIPs/13NNN-WGB.md


I'm well aware, that one is a much needed enhancement. But it's a 
long ways away, and would have to be adapted to through quite a 
process of its own.
Jun 20 2020
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/20/20 8:30 AM, Stanislav Blinov wrote:

 On Saturday, 20 June 2020 at 11:20:13 UTC, Petar Kirov [ZombineDev] wrote:
 

 As far as I know, back when the ranges API was worked on postblit 
 ctors didn't work reliably, so .save() was the only option.


 
 Funny that, at the moment it's the copy ctors that aren't working 
 reliably (e.g. they're not working at all with arrays). :)
 

 If it had worked, we could require that non-forward ranges are 
 non-copyable.


 
 I can imagine this would be quite some work to adapt all of Phobos to 
 *that*. I mean, things like
 
 auto rem = makeSomeInputRange.find!pred;
 auto flt = makeSomeInputRange.filter!pred;
 
 Those would not compile. They could be made to compile by `move`ing the 
 argument for the return. But then you still won't be able to pass those 
 results around, unless via a refRange or `move`.


You shouldn't need move for rvalues, which those should be. Algorithms 
that support non-forward ranges should be able to cope with using move 
where required on their parameters.

There have been a few libraries released that use non-copyable structs 
to model things that making copies of will prove problematic. For 
instance, file handles. And people seem to be able to use these 
reasonably well.

But the larger point is that true input-only ranges are rare. The only 
problem is for classes, since you cannot invoke a copy constructor on those.

It would be a drastic change, I don't see it happening. But the status 
quo of save() is pretty bad as well.

-Steve
Jun 20 2020
next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 20 June 2020 at 18:26:43 UTC, Steven Schveighoffer 
wrote:

 On 6/20/20 8:30 AM, Stanislav Blinov wrote:



 I can imagine this would be quite some work to adapt all of 
 Phobos to *that*. I mean, things like
 
 auto rem = makeSomeInputRange.find!pred;
 auto flt = makeSomeInputRange.filter!pred;
 
 Those would not compile. They could be made to compile by 
 `move`ing the argument for the return. But then you still 
 won't be able to pass those results around, unless via a 
 refRange or `move`.


 You shouldn't need move for rvalues, which those should be. 
 Algorithms that support non-forward ranges should be able to 
 cope with using move where required on their parameters.


`find` returns the remainder of the range, i.e it's (pseudocode):

auto find(alias pred,R)(R r)
{
     while (!r.empty && !pred(r.front))
         r.popFront;
     return r;
}

There should be a `move` at that return. There's no NRVO, the 
compiler makes a copy there.  Which, of course, it wouldn't be 
able to if R was non-copyable :) It *may* become a move under 
Walter's proposal:

auto find(alias pred,R)(return R r)
{
     while (!r.empty && !pred(r.front))
         r.popFront;
     return r; // return what's called a `move ref` in the proposal
}

but until then?..


 There have been a few libraries released that use non-copyable 
 structs to model things that making copies of will prove 
 problematic. For instance, file handles. And people seem to be 
 able to use these reasonably well.


We're talking ranges here, not file handles :) Most algorithms in 
Phobos take ranges by value.


 But the larger point is that true input-only ranges are rare. 
 The only problem is for classes, since you cannot invoke a copy 
 constructor on those.

 It would be a drastic change, I don't see it happening. But the 
 status quo of save() is pretty bad as well.


*Why* is it bad?
Jun 20 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/20/20 3:16 PM, Stanislav Blinov wrote:

 On Saturday, 20 June 2020 at 18:26:43 UTC, Steven Schveighoffer wrote:

 On 6/20/20 8:30 AM, Stanislav Blinov wrote:


 

 I can imagine this would be quite some work to adapt all of Phobos to 
 *that*. I mean, things like

 auto rem = makeSomeInputRange.find!pred;
 auto flt = makeSomeInputRange.filter!pred;

 Those would not compile. They could be made to compile by `move`ing 
 the argument for the return. But then you still won't be able to pass 
 those results around, unless via a refRange or `move`.


 You shouldn't need move for rvalues, which those should be. Algorithms 
 that support non-forward ranges should be able to cope with using move 
 where required on their parameters.


 
 `find` returns the remainder of the range, i.e it's (pseudocode):
 
 auto find(alias pred,R)(R r)
 {
      while (!r.empty && !pred(r.front))
          r.popFront;
      return r;
 }
 
 There should be a `move` at that return. There's no NRVO, the compiler 
 makes a copy there.  Which, of course, it wouldn't be able to if R was 
 non-copyable :)


Yeah, it needs to use move. But the usage doesn't change, which is what 
I thought you were referring to.


 It *may* become a move under Walter's proposal:
 
 auto find(alias pred,R)(return R r)
 {
      while (!r.empty && !pred(r.front))
          r.popFront;
      return r; // return what's called a `move ref` in the proposal
 }
 
 but until then?..


You'd have to modify Phobos.


 

 There have been a few libraries released that use non-copyable structs 
 to model things that making copies of will prove problematic. For 
 instance, file handles. And people seem to be able to use these 
 reasonably well.


 
 We're talking ranges here, not file handles :) Most algorithms in Phobos 
 take ranges by value.


Which means that you have to destroy the original, because it's no 
longer valid.

Today, you do:

auto otherRange = inputRange.find(something);

use(inputRange); // invalid, but compiler lets you. And it probably 
messes up otherRange too.


 

 But the larger point is that true input-only ranges are rare. The only 
 problem is for classes, since you cannot invoke a copy constructor on 
 those.

 It would be a drastic change, I don't see it happening. But the status 
 quo of save() is pretty bad as well.


 
 *Why* is it bad?


Because there's no enforcement. You can copy most forward ranges without 
using save, and it works EXACTLY the same. In fact, most save 
implementations are {return this;}. If you ever use a forward range that 
requires the use of .save to actually copy it, you will have problems.

So it essentially promotes generic code that is incorrect, but works 
fine for most cases. I'm positive there are still cases like this in 
Phobos. It's very easy and effortless to make a copy without using .save.

When's the last time you saw:

foreach(x; someLvalueRange.save)

Because if you don't see that, for a forward range, it's an incorrect usage.

-Steve
Jun 20 2020
parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 20 June 2020 at 20:09:56 UTC, Steven Schveighoffer 
wrote:

 On 6/20/20 3:16 PM, Stanislav Blinov wrote:

 There should be a `move` at that return. There's no NRVO, the 
 compiler makes a copy there.  Which, of course, it wouldn't be 
 able to if R was non-copyable :)


 Yeah, it needs to use move. But the usage doesn't change, which 
 is what I thought you were referring to.


The usage does change. Because then (i.e. if input ranges were to 
be made non-copyable):

auto r = makeInputRange;
auto rem = r.find!pred;

...no longer compiles. And *that* will be all over Phobos, which 
is my original point about required massive changes. Which, as it 
turns out, may not be that bad of a thing in principle (though, 
of course, going over all parts of Phobos that touch ranges to 
fix them would be a huge endeavor).


 It *may* become a move under Walter's proposal:
 but until then?..


 You'd have to modify Phobos.


Exactly.


 There have been a few libraries released that use 
 non-copyable structs to model things that making copies of 
 will prove problematic. For instance, file handles. And 
 people seem to be able to use these reasonably well.


 
 We're talking ranges here, not file handles :) Most algorithms 
 in Phobos take ranges by value.





 Which means that you have to destroy the original, because it's 
 no longer valid.

 Today, you do:

 auto otherRange = inputRange.find(something);

 use(inputRange); // invalid, but compiler lets you. And it 
 probably messes up otherRange too.


Yeah, that's a good point, thanks.


 But the larger point is that true input-only ranges are rare. 
 The only problem is for classes, since you cannot invoke a 
 copy constructor on those.

 It would be a drastic change, I don't see it happening. But 
 the status quo of save() is pretty bad as well.


 
 *Why* is it bad?


 Because there's no enforcement. You can copy most forward 
 ranges without using save, and it works EXACTLY the same. In 
 fact, most save implementations are {return this;}. If you ever 
 use a forward range that requires the use of .save to actually 
 copy it, you will have problems.

 So it essentially promotes generic code that is incorrect, but 
 works fine for most cases. I'm positive there are still cases 
 like this in Phobos. It's very easy and effortless to make a 
 copy without using .save.

 When's the last time you saw:

 foreach(x; someLvalueRange.save)

 Because if you don't see that, for a forward range, it's an 
 incorrect usage.


In that case, making input ranges non-copyable *may* indeed be a 
solution.
Jun 20 2020
prev sibling next sibling parent reply Jon Degenhardt <jond noreply.com> writes:
On Saturday, 20 June 2020 at 18:26:43 UTC, Steven Schveighoffer 
wrote:

 But the larger point is that true input-only ranges are rare. 
 The only problem is for classes, since you cannot invoke a copy 
 constructor on those.


Interesting discussion. Could you expand on this comment? Several 
people have mentioned this.

I write my own input ranges somewhat regularly. I've never had 
the need to make them forward ranges. However, the typical reason 
for creating a range is because I have application specific data 
that I want to iterate over (and usually construct) lazily. Input 
ranges are very convenient way to do this. I do end up making 
many of them reference ranges though.

So, I'm wondering if its really that input-only ranges are rare, 
or if it's that the number of algorithms that can be used on 
input-only ranges is small. Or perhaps I'm not quite grokking the 
distinction between a "true" input-only range and one that 
satisfies isInputRange, but none of the other range primitive 
tests.

--Jon
Jun 20 2020
next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Saturday, 20 June 2020 at 19:23:55 UTC, Jon Degenhardt wrote:

 On Saturday, 20 June 2020 at 18:26:43 UTC, Steven Schveighoffer 
 wrote:

 But the larger point is that true input-only ranges are rare. 
 The only problem is for classes, since you cannot invoke a 
 copy constructor on those.


 Interesting discussion. Could you expand on this comment? 
 Several people have mentioned this.

 I write my own input ranges somewhat regularly. I've never had 
 the need to make them forward ranges. However, the typical 
 reason for creating a range is because I have application 
 specific data that I want to iterate over (and usually 
 construct) lazily. Input ranges are very convenient way to do 
 this. I do end up making many of them reference ranges though.

 So, I'm wondering if its really that input-only ranges are 
 rare, or if it's that the number of algorithms that can be used 
 on input-only ranges is small. Or perhaps I'm not quite 
 grokking the distinction between a "true" input-only range and 
 one that satisfies isInputRange, but none of the other range 
 primitive tests.

 --Jon


Could you easily make your ranges into forward ranges if you 
wanted to? That is, if you copy one of them, and then iterate the 
original, does the copy remain valid? If so, they're not "true" 
input ranges.

A "true" input range is something like a file handle, where you 
couldn't implement `save` even if you wanted to--iterating one 
copy automatically invalidates all the others.
Jun 20 2020
parent Jon Degenhardt <jond noreply.com> writes:
On Saturday, 20 June 2020 at 19:35:07 UTC, Paul Backus wrote:

 On Saturday, 20 June 2020 at 19:23:55 UTC, Jon Degenhardt wrote:

 On Saturday, 20 June 2020 at 18:26:43 UTC, Steven 
 Schveighoffer wrote:

 But the larger point is that true input-only ranges are rare. 
 The only problem is for classes, since you cannot invoke a 
 copy constructor on those.


 Interesting discussion. Could you expand on this comment? 
 Several people have mentioned this.

 I write my own input ranges somewhat regularly. I've never had 
 the need to make them forward ranges. However, the typical 
 reason for creating a range is because I have application 
 specific data that I want to iterate over (and usually 
 construct) lazily. Input ranges are very convenient way to do 
 this. I do end up making many of them reference ranges though.

 So, I'm wondering if its really that input-only ranges are 
 rare, or if it's that the number of algorithms that can be 
 used on input-only ranges is small. Or perhaps I'm not quite 
 grokking the distinction between a "true" input-only range and 
 one that satisfies isInputRange, but none of the other range 
 primitive tests.

 --Jon


 Could you easily make your ranges into forward ranges if you 
 wanted to? That is, if you copy one of them, and then iterate 
 the original, does the copy remain valid? If so, they're not 
 "true" input ranges.

 A "true" input range is something like a file handle, where you 
 couldn't implement `save` even if you wanted to--iterating one 
 copy automatically invalidates all the others.


Ah, thanks for this clarification. I'd have to go back and look 
at all the different ranges I've written, but I'd say there are a 
fair number in both camps. Now I'm wondering if there's a 
correlation with the use of reference ranges. That is, have I 
used reference ranges when the range falls into the "true" input 
range set. The answer may be yes.
Jun 20 2020
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/20/20 3:23 PM, Jon Degenhardt wrote:

 On Saturday, 20 June 2020 at 18:26:43 UTC, Steven Schveighoffer wrote:

 But the larger point is that true input-only ranges are rare. The only 
 problem is for classes, since you cannot invoke a copy constructor on 
 those.


 
 Interesting discussion. Could you expand on this comment? Several people 
 have mentioned this.


There are a few "base" ranges, like arrays, and data structure ranges. 
The one common true "input-only" range is a stream-based range (like 
File.byLine).

Other than that, wrapping ranges should implement the primitives that 
their dependencies do.

I'd say arrays are the most common range, and used for almost 
everything. Even with file streams, most of the time, you don't use the 
stream as a range, but use algorithms on the buffered data (which is an 
array).

An input range is generally useful in foreach, and almost nothing else.


 
 I write my own input ranges somewhat regularly. I've never had the need 
 to make them forward ranges. However, the typical reason for creating a 
 range is because I have application specific data that I want to iterate 
 over (and usually construct) lazily. Input ranges are very convenient 
 way to do this. I do end up making many of them reference ranges though.


Many people do not bother adding ".save" because it's an extra thing, 
and copying works fine even if you don't declare save.


 
 So, I'm wondering if its really that input-only ranges are rare, or if 
 it's that the number of algorithms that can be used on input-only ranges 
 is small. Or perhaps I'm not quite grokking the distinction between a 
 "true" input-only range and one that satisfies isInputRange, but none of 
 the other range primitive tests.


An input range cannot be (correctly) iterated more than once, even if 
you make copies of it.

-Steve
Jun 20 2020
parent reply Jon Degenhardt <jond noreply.com> writes:
On Saturday, 20 June 2020 at 20:17:29 UTC, Steven Schveighoffer 
wrote:

 On 6/20/20 3:23 PM, Jon Degenhardt wrote:

 On Saturday, 20 June 2020 at 18:26:43 UTC, Steven 
 Schveighoffer wrote:

 But the larger point is that true input-only ranges are rare. 
 The only problem is for classes, since you cannot invoke a 
 copy constructor on those.


 
 Interesting discussion. Could you expand on this comment? 
 Several people have mentioned this.


 There are a few "base" ranges, like arrays, and data structure 
 ranges. The one common true "input-only" range is a 
 stream-based range (like File.byLine).


Thanks for the detailed reply, it's very helpful and I'm finding 
the discussion useful.

Another potentially useful example is a stream of random numbers, 
where it may be undesirable to allow the state of the random 
number stream to be copied.


 An input range is generally useful in foreach, and almost 
 nothing else.


Here I might disagree a bit. Nearly all the algorithms in 
std.algorithm.iteration (map, filter, fold, etc.) operate on 
input-only ranges. These are important classes of operations. 
They are largely alternate forms of foreach, so likely this is 
more clarification than disagreement.

Related thought: One-time iteration of a stream is common in many 
of the apps I write. I tend to think of such streams as logically 
input-only. Whether it's valid/legal to copy the stream at 
implementation level is a different question. More recently I've 
finding reasons to enforce at most once iteration, even if the 
underlying data structures (like an array) don't care.

--Jon
Jun 20 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/20 5:12 PM, Jon Degenhardt wrote:

 Nearly all the algorithms in std.algorithm.iteration (map, filter, fold, 
 etc.) operate on input-only ranges.


Yah, one good experiment would be to implement these using alternate 
simpler APIs and see how they work.

You can infer a lot by just writing code and "smell" it. I know byLine 
didn't smell quite right.
Jun 20 2020
parent Jon Degenhardt <jond noreply.com> writes:
On Sunday, 21 June 2020 at 00:15:35 UTC, Andrei Alexandrescu 
wrote:

 On 6/20/20 5:12 PM, Jon Degenhardt wrote:

 Nearly all the algorithms in std.algorithm.iteration (map, 
 filter, fold, etc.) operate on input-only ranges.


 Yah, one good experiment would be to implement these using 
 alternate simpler APIs and see how they work.

 You can infer a lot by just writing code and "smell" it. I know 
 byLine didn't smell quite right.


Yeah, that would be a nice focused set to try it out on.
Jun 21 2020
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/20 3:23 PM, Jon Degenhardt wrote:

 On Saturday, 20 June 2020 at 18:26:43 UTC, Steven Schveighoffer wrote:

 But the larger point is that true input-only ranges are rare. The only 
 problem is for classes, since you cannot invoke a copy constructor on 
 those.


 
 Interesting discussion. Could you expand on this comment? Several people 
 have mentioned this.
 
 I write my own input ranges somewhat regularly. I've never had the need 
 to make them forward ranges. However, the typical reason for creating a 
 range is because I have application specific data that I want to iterate 
 over (and usually construct) lazily. Input ranges are very convenient 
 way to do this. I do end up making many of them reference ranges though.
 
 So, I'm wondering if its really that input-only ranges are rare, or if 
 it's that the number of algorithms that can be used on input-only ranges 
 is small. Or perhaps I'm not quite grokking the distinction between a 
 "true" input-only range and one that satisfies isInputRange, but none of 
 the other range primitive tests.


Numbers are kinda in your favor. Without getting anywhere near 
scientific about it:

git grep isInputRange | wc -l
      601
git grep isForwardRange | wc -l
      369
git grep isBidirectionalRange | wc -l
      167
git grep isRandomAccessRange | wc -l
      285
Jun 20 2020
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/20 2:26 PM, Steven Schveighoffer wrote:

 It would be a drastic change, I don't see it happening.


I totally see it happening if we go with versioning.

D has this good module system, so it should work real nice. To the 
extent it doesn't, it would point to opportunities for improvement.

The stalemate we're in has taken long enough.
Jun 20 2020
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/20 6:03 AM, Petar Kirov [ZombineDev] wrote:

 On Saturday, 20 June 2020 at 09:49:38 UTC, Stanislav Blinov wrote:

 On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:

 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu via 
 Digitalmars-d wrote: [...]

 One good goal for std.v2020 would be to forego autodecoding throughout.


 [...]

 Another could be to fix up the range API -- i.e, reconsider the 
 ugliness that is .save, now that D has copy ctors.


 How do you see that? Fundamentally, what have copy ctors changed in 
 this regard?


 
 I suppose that postblit has had various implementation and language 
 design issues (which copy constructors address), which prevented them 
 from being a reliable alternative to save(). This and probably also 
 class support.


That's not the problem. There'd need to be an API distinction between 
the two that is introspectable. Many input ranges should be copyable, 
which makes them indistinguishable from forward range by means of copy ctor.
Jun 20 2020
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/20 5:49 AM, Stanislav Blinov wrote:

 On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:

 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu via 
 Digitalmars-d wrote: [...]

 One good goal for std.v2020 would be to forego autodecoding throughout.


 [...]

 Another could be to fix up the range API -- i.e, reconsider the 
 ugliness that is .save, now that D has copy ctors.


 
 How do you see that? Fundamentally, what have copy ctors changed in this 
 regard?


I've been thinking of this for a while. The fact that input ranges must 
buffer one element (i.e. make front() callable several times) has been a 
gauche choice.

Input ranges should have only one API:

bool fetchNext(T& target);

Fill the user-provided target with the next element and return true. At 
the end of the range, return false and leave the target alone.

I'd considered this design back in the day but I was annoyed that it 
required a copy for ranges that in fact are better than input ranges 
(e.g. arrays etc).

It's still a disadvantage, but over the years I realized there are not 
that many algorithms that only work on input ranges so we could make-do. 
Of those for which efficiency is a concern, they could easily be 
specialized for forward ranges.
Jun 20 2020
next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 20 June 2020 at 14:51:32 UTC, Andrei Alexandrescu 
wrote:

 On 6/20/20 5:49 AM, Stanislav Blinov wrote:

 On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:

 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu 
 via Digitalmars-d wrote: [...]

 One good goal for std.v2020 would be to forego autodecoding 
 throughout.


 [...]

 Another could be to fix up the range API -- i.e, reconsider 
 the ugliness that is .save, now that D has copy ctors.


 
 How do you see that? Fundamentally, what have copy ctors 
 changed in this regard?


 I've been thinking of this for a while. The fact that input 
 ranges must buffer one element (i.e. make front() callable 
 several times) has been a gauche choice.


I think there's a bit of conflation going on here. Being able to 
call front() several times does not necessitate the *range* being 
buffering. After all, the range, generally speaking, need not 
store any data at all. Buffering comes as a necessity for some 
*data sources*, which are either slow to fetch the data, or 
simply don't have it yet, not unlike the one you're showing below:


 Input ranges should have only one API:

 bool fetchNext(T& target);

 Fill the user-provided target with the next element and return 
 true. At the end of the range, return false and leave the 
 target alone.


For anything other than PODs, this API is less efficient and 
forces the callers to, effectively, do busy work (i.e. 
boilerplate just to facilitate language plumbing).

while (!range.empty)
{
     dataStructure.emplace(range.front); // *
     range.popFront();
}

As far as the range API itself is concerned, the above line (*), 
under Walter's "moving" proposal, is *at most* a constructor call 
and a move-construct (or just the constructor call, in a good 
optimizing compiler with the right data structure 
implementation), and *no* (zero!) destructor calls. Mind you, the 
`front` need not, necessarily, be returning a copy. Consider e.g. 
a range consuming a socket: it'd just construct the element out 
of buffered data and return it with NRVO (IOW, construct it 
directly on caller's stack, or, potentially, straight in the 
`dataStructure`).

Versus this:

while (true)
{
     ElementType tmp = ElementType.init;
     if (range.fetchNext(tmp)) // That's a copy-assign, i.e. a 
(redundant) dtor + copy
         dataStructure.emplace(move(tmp)); // one (or two) 
move-constructs
     else
         break;
} // redundant dtor

I am by no means an expert on compilers, but something tells me 
that eliding redundant calls and copies from *that* isn't a task 
for the light-hearted.

I'll take a "buffering" API over that any day. Not that the 
`fetchNext` API doesn't have its uses (it absolutely does), I 
just don't think it's a good candidate for ranges.
Jun 20 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/20 12:41 PM, Stanislav Blinov wrote:

 Being able to call front() several times does not necessitate the 
 *range* being buffering.


If one calls front() twice without advancing the range, where does the 
range return the value from?

Input ranges must buffer one element at least (input iterators in STL, 
too). This has been a source of annoyance in implementing them.
Jun 20 2020
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Sunday, 21 June 2020 at 00:05:01 UTC, Andrei Alexandrescu 
wrote:

 On 6/20/20 12:41 PM, Stanislav Blinov wrote:

 Being able to call front() several times does not necessitate 
 the *range* being buffering.


 If one calls front() twice without advancing the range, where 
 does the range return the value from?


The most primitive example? The range is lazy and builds the 
element on every call. But I believe I've tried to make the 
distinction clear: *range* itself isn't necessarily buffering. 
Whatever it's iterating over may have to.

And, in case that wasn't clear either, I understand perfectly 
where you're coming from, since typical input ranges do indeed 
have to memoize that nasty first element upon initialization (and 
since they have to do that, they have to repeat that for every 
element thenceforth). Thing is, the input stream

`bool fetchNext(ref T);`

does not solve this problem, it merely shrugs it off unto the 
caller, with precarious consequences, as demonstrated.
Jun 20 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/20 8:38 PM, Stanislav Blinov wrote:

 On Sunday, 21 June 2020 at 00:05:01 UTC, Andrei Alexandrescu wrote:

 On 6/20/20 12:41 PM, Stanislav Blinov wrote:

 Being able to call front() several times does not necessitate the 
 *range* being buffering.


 If one calls front() twice without advancing the range, where does the 
 range return the value from?


 
 The most primitive example? The range is lazy and builds the element on 
 every call. But I believe I've tried to make the distinction clear: 
 *range* itself isn't necessarily buffering. Whatever it's iterating over 
 may have to.
 
 And, in case that wasn't clear either, I understand perfectly where 
 you're coming from, since typical input ranges do indeed have to memoize 
 that nasty first element upon initialization (and since they have to do 
 that, they have to repeat that for every element thenceforth). Thing is, 
 the input stream
 
 `bool fetchNext(ref T);`
 
 does not solve this problem, it merely shrugs it off unto the caller, 
 with precarious consequences, as demonstrated.


I appreciate there's no shortage of people who teach my design and my 
code back to me. I honestly do, a lot.

It's unclear that much of anything, was demonstrated. Was the better 
alternative to make input ranges noncopyable? It only takes a couple of 
hours of coding a bit of that up to see it's quite onerous.
Jun 21 2020
next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Sunday, 21 June 2020 at 14:47:55 UTC, Andrei Alexandrescu 
wrote:


 I appreciate there's no shortage of people who teach my design 
 and my code back to me. I honestly do, a lot.


And I, in turn, appreciate that you nicked one paragraph and 
ignored (I assume dismissed) everything else. Very motivating.


 It's unclear that much of anything, was demonstrated. Was the 
 better alternative to make input ranges noncopyable?


No. No. No. It wasn't an alternative to your API. We weren't even 
discussing a completely alternative API until you threw it in.
I compared of two APIs in terms of operations. But "nothing was 
demonstrated". OK! Anyway, the whole discussion was going around 
the existing API, and either:

- input ranges non-copyable and forward ranges are, or
- status quo (i.e. save() for forward ranges)

I didn't catch on the distinction initially, when H.S. brought up 
the save(); now I do.

TLDR for all of the below - under either API, input ranges being 
non-copyable is a much better choice.

Input ranges, by nature being one-pass, *should not be copyable*. 
You can't do anything (good) with a copy, and have to invest into 
implementing a copy that won't bite. If you're giving such range 
away - you're giving it *away*, to someone else to consume. It 
being copyable only means that you're leaving for yourself a 
mutable reference to state that you shouldn't touch again. When 
you need the remainder back - your callee will move it back.

You mentioned the "smell" of ByLine. A good deal of that "smell" 
emanates from its copy-ability. I mean, disabling a constructor 
versus reference-counted internals - which plate is heavier?... 
Using either API, that smell will stay if it is to remain 
copyable.

Most of the time you'd be using input ranges as rvalues. In the 
remaining cases the compiler will give you an error if you try to 
copy - and that's a good error.


 It only takes a couple of hours of coding a bit of that up to 
 see it's quite onerous.


It isn't. Bugs in the language and Phobos aside - it's one `move` 
away.

Here, a few Phobos algorithms implemented with the `fetchNext` 
API:

https://gist.github.com/radcapricorn/d76d29c6df6fa822d7889e799937f39d

The good? An actual source range (ByLine) is 
dumb-as-a-cork-simple. No buffering, no ref-counted internals: 
read, slice, rinse and repeat. The bad? Well, you can see for 
yourself.
Jun 21 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 6/21/20 1:43 PM, Stanislav Blinov wrote:

 Input ranges, by nature being one-pass, *should not be copyable*. You 
 can't do anything (good) with a copy, and have to invest into 
 implementing a copy that won't bite. If you're giving such range away - 
 you're giving it *away*, to someone else to consume. It being copyable 
 only means that you're leaving for yourself a mutable reference to state 
 that you shouldn't touch again. When you need the remainder back - your 
 callee will move it back.


Good arguments, no doubt, but a long experience with noncopyable C++ 
objects suggests that defining such types need to be approached with 
trepidation as they are very cumbersome to use. (Any type containing a 
noncopyable type as a member becomes itself noncopyable; this is not 
something that we can inflict lightly on the casual users of input 
ranges. (To wit: no input streams or iterators in C++ are noncopyable, 
although the same argument would apply to them as well.)

Also, it is not unheard of to have two input ranges fed from the same 
source with the obvious semantics (whoever calls functions to get more 
data will get the data and push the cursor further). True, buffering is 
an issue (what if two copies have each their own buffers?) but that's an 
engineering problem, not one of principles.

It is quite clear to me that we can't propose a design with noncopyable 
input ranges without effectively making them pariahs that everybody will 
take pains to use and do their best to avoid.
Jun 22 2020
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Monday, 22 June 2020 at 16:03:54 UTC, Andrei Alexandrescu 
wrote:

 On 6/21/20 1:43 PM, Stanislav Blinov wrote:

 Input ranges, by nature being one-pass, *should not be 
 copyable*...





 Good arguments, no doubt, but a long experience with 
 noncopyable C++ objects suggests that defining such types need 
 to be approached with trepidation as they are very cumbersome 
 to use.


In C++ - traditionally they were cumbersome, even to define, 
until move came along. In D though? The biggest annoying obstacle 
is Phobos making lots of redundant copies, needlessly prohibiting 
the use of non-copyables. That, however, is not a fault of 
non-copyables, but of Phobos.
The GC limits the use of non-copyables for sure, though that's 
largely irrelevant to ranges themselves, esp. considering that 
classes indeed shouldn't ever be ranges.


 (Any type containing a noncopyable type as a member becomes 
 itself noncopyable;


And that is *good*.


 this is not something that we can inflict lightly on the casual 
 users of input ranges.


That is exactly what we *should* inflict on "casual" users. 
Compile-time errors. For trying to casually write bugs.


 (To wit: no input streams or iterators in C++ are noncopyable, 
 although the same argument would apply to them as well.)


An input range being non-copyable is immensely beneficial - it 
would statically catch a good deal of problems. It would 
*alleviate* the need of added complexity in implementations. 
Looking back at the ByLine again: RefCounted internals just to 
remain copyable. I don't think appeasing "casual users" by 
over-engineering is a good compromise.
Non-copyables in D are easy to write, easy to use *correctly*, 
and difficult to use *incorrectly* (won't compile, after all).


 Also, it is not unheard of to have two input ranges fed from 
 the same source with the obvious semantics (whoever calls 
 functions to get more data will get the data and push the 
 cursor further). True, buffering is an issue (what if two 
 copies have each their own buffers?) but that's an engineering 
 problem, not one of principles.


That would be different instances of input ranges over the same 
source, not copies of the same range carrying unnecessary state 
and complexity (over-engineered solutions to the buffering 
issue). We're never consuming the same range *simultaneously* in 
different places. With an uncopyable, we'd just have to give it 
to a consumer and then have them give the remainder back to us.


 It is quite clear to me that we can't propose a design with 
 noncopyable input ranges without effectively making them 
 pariahs that everybody will take pains to use and do their best 
 to avoid.


Then we should propose a design that is not painful to use:

// error: `input` cannot be copied
// auto data = input.filter!somehow.map!something.array;
// Ok:
auto data = input.move.filter!somehow.map!something.array;

If we need partial consumption, i.e. preservation of the 
remainder, terminal primitives can give it back to us (after all, 
wrapping range is holding onto it):

auto data = 
input.move.filter!somehow.take(someAmount).map!something.array(input);

The implementation of such `array` would look like this:

auto array(Range, Remainder)(Range range, ref Remainder remainder)
if (isInputRange!Range && is(typeof(range.source) == Remainder))
{
     import std.array : appender;
     auto builder = appender!(ElementType!Range[]);
     // this is assuming current API, substitute with whatever 
replacement API
     while (!range.empty)
     {
         builder.put(range.front);
         range.popFront();
     }
     move(range.source, remainder);
     return builder[];
}

I wouldn't say this qualifies as "painful", to author or use. 
Authors of ranges would be writing ranges, and not the machinery 
required to keep them copyable. Users of ranges would be using 
ranges, and not the hidden machinery required to keep them 
copyable.
Jun 24 2020
parent reply Paul Backus <snarwin gmail.com> writes:
On Wednesday, 24 June 2020 at 11:49:11 UTC, Stanislav Blinov 
wrote:

 On Monday, 22 June 2020 at 16:03:54 UTC, Andrei Alexandrescu 
 wrote:

 It is quite clear to me that we can't propose a design with 
 noncopyable input ranges without effectively making them 
 pariahs that everybody will take pains to use and do their 
 best to avoid.


 Then we should propose a design that is not painful to use:

 // error: `input` cannot be copied
 // auto data = input.filter!somehow.map!something.array;
 // Ok:
 auto data = input.move.filter!somehow.map!something.array;

 If we need partial consumption, i.e. preservation of the 
 remainder, terminal primitives can give it back to us (after 
 all, wrapping range is holding onto it):

 auto data = 
 input.move.filter!somehow.take(someAmount).map!something.array(input);


IMO if the user has to manually call `move`, you have already 
failed at usability.

It may seem "easy" or "obvious" to you and I, but for beginners, 
this is going to be a huge stumbling block. You write some code 
that looks like it should obviously work, you get a mysterious 
error message that "std.algorithm.whatever!(some, args).Result is 
not copyable because it is annotated with  disable", and the 
solution is that you have to add ".move" to your code? Can you 
imagine having to explain that to someone in the Learn forum? I 
don't think I could do it with a straight face--it's too absurd.

The only way non-copyable input ranges can work is if the 
compiler is able to implicitly move them in cases like the above. 
In other words, we would need something like Walter's "Copying, 
Moving, and Forwarding" DIP [1].

[1] 
https://github.com/WalterBright/DIPs/blob/13NNN-WGB.md/DIPs/13NNN-WGB.md
Jun 24 2020
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Wednesday, 24 June 2020 at 13:14:22 UTC, Paul Backus wrote:

 On Wednesday, 24 June 2020 at 11:49:11 UTC, Stanislav Blinov 
 wrote:

 On Monday, 22 June 2020 at 16:03:54 UTC, Andrei Alexandrescu 
 wrote:

 It is quite clear to me that we can't propose a design with 
 noncopyable input ranges without effectively making them 
 pariahs that everybody will take pains to use and do their 
 best to avoid.


 Then we should propose a design that is not painful to use:

 // error: `input` cannot be copied
 // auto data = input.filter!somehow.map!something.array;
 // Ok:
 auto data = input.move.filter!somehow.map!something.array;

 If we need partial consumption, i.e. preservation of the 
 remainder, terminal primitives can give it back to us (after 
 all, wrapping range is holding onto it):

 auto data = 
 input.move.filter!somehow.take(someAmount).map!something.array(input);


 IMO if the user has to manually call `move`, you have already 
 failed at usability.

 It may seem "easy" or "obvious" to you and I, but for 
 beginners, this is going to be a huge stumbling block.


Libraries shan't be for "beginners" nor "advanced users". And 
that use case is already a non-beginner. Beginners would write 
this:

auto data = 
createSomeInput(args).filter!somehow.map!something.array;

Create input, consume input, with no explicit moves in sight. 
Piecemeal consumption of an input range is something an 
"advanced" user would be doing :)

I am also of the opinion that a huge reminder "YOU SHALL NOT COPY 
UNLESS YOU MUST" needs to be present on every page of any 
introductory (and not only) literature, and not just for D. The 
problem with lax copying extends far outside the realm of ranges.

struct CopyCount
{
     int count;
     this(ref typeof(this) other) { count = other.count + 1; }
}

CopyCount cc;
writeln(cc);

In my book, that should print CopyCount(0) (i.e. you print that 
which you can parse and get the original). That's not what Phobos 
writeln would print though. It won't even print CopyCount(1).

People need to be taught to not squander their values, and how 
not to.


 You write some code that looks like it should obviously work, 
 you get a mysterious error message that 
 "std.algorithm.whatever!(some, args).Result is not copyable 
 because it is annotated with  disable"


Yup, because in that hypothetical universe you're using a type 
that is documented to be an input range, a category of ranges 
which is documented to be non-copyable, which means you just 
wrote what otherwise would've been a bug, but the compiler 
stopped you.


 and the solution is that you have to add ".move" to your code? 
 Can you imagine having to explain that to someone in the Learn 
 forum? I don't think I could do it with a straight face--it's 
 too absurd.


Nothing to it - you point such user to the documentation of 
ranges which states that input ranges are non-copyable, explains 
why they're non-copyable, and has usage examples for rvalues and 
lvalues.
You're aware of a recent question in .learn there, where the user 
attempted to iterate a non-copyable range (the thread which 
turned into another prolonged discussion: 
https://forum.dlang.org/post/kpncjzadrwpvxupsdmle forum.dlang.org). A range
which, by the way, was an input range. The explanations quickly went into
"common pitfalls" territory. Jonathan M Davis' response was of particular
interest: "In general, you should never use a range after it's been copied
unless you know exactly what type of range you're dealing with and what its
copying behavior is. If you want an independent copy, you need to use save."
I don't think explaining that to a "beginner" is more practical 
than explaining that input ranges are consumed once and may be 
yielding values that are only valid between iterations.


 The only way non-copyable input ranges can work is if the 
 compiler is able to implicitly move them in cases like the 
 above. In other words, we would need something like Walter's 
 "Copying, Moving, and Forwarding" DIP [1].

 [1] 
 https://github.com/WalterBright/DIPs/blob/13NNN-WGB.md/DIPs/13NNN-WGB.md


The way that proposal exists at the moment, it won't help 
practically whenever you're not at "last use" (i.e. the second 
case - wanting to keep the lvalue for the remainder). Compiler 
will want to copy then. It'll certainly help for the first case 
though, as well as general implementation of ranges.
Jun 24 2020
parent reply Paul Backus <snarwin gmail.com> writes:
On Wednesday, 24 June 2020 at 15:26:40 UTC, Stanislav Blinov 
wrote:

 Libraries shan't be for "beginners" nor "advanced users". And 
 that use case is already a non-beginner. Beginners would write 
 this:

 auto data = 
 createSomeInput(args).filter!somehow.map!something.array;

 Create input, consume input, with no explicit moves in sight. 
 Piecemeal consumption of an input range is something an 
 "advanced" user would be doing :)


Ok. Now they refactor their code:

auto input = createSomeInput(args);
auto data = input.filter!somehow.map!something.array;

Kaboom! Compile error. This is why Walter's proposal is relevant 
here.


 You're aware of a recent question in .learn there, where the 
 user attempted to iterate a non-copyable range (the thread 
 which turned into another prolonged discussion: 
 https://forum.dlang.org/post/kpncjzadrwpvxupsdmle forum.dlang.org). A range
which, by the way, was an input range. The explanations quickly went into
"common pitfalls" territory. Jonathan M Davis' response was of particular
interest: "In general, you should never use a range after it's been copied
unless you know exactly what type of range you're dealing with and what its
copying behavior is. If you want an independent copy, you need to use save."


To be clear: I agree with you *in principle* that pure input 
ranges should be non-copyable. It's just that currently, 
non-copyable types are rather cumbersome to work with in D, and 
IMO it would be a step backwards in terms of usability to force 
them on users of std.algorithm and std.range.

The solution is to make working with non-copyable types more 
ergonomic, and then make pure input ranges non-copyable.

This will also improve the signal-to-noise ratio of compile 
errors: with automatic move-on-last-use, when the compiler 
complains about copying a range, you can be sure you have 
actually made a mistake somewhere in your logic, rather than just 
forgetting to type ".move".
Jun 24 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/24/20 11:55 AM, Paul Backus wrote:

 To be clear: I agree with you *in principle* that pure input ranges 
 should be non-copyable.


By what principle two input ranges should absolutely never use the same 
data feed?
Jun 24 2020
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
I'm barely reading this thread, but could  live be useful with 
input ranges too?
Jun 24 2020
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/24/20 12:25 PM, Adam D. Ruppe wrote:

 I'm barely reading this thread, but could  live be useful with input 
 ranges too?


I'm not sure, but with a no-quarter-given approach to copying input 
ranges, simple tasks like "feed input from this file, or from stdin, and 
output to this other file or to stdout" mutate from trivial into little 
research projects (btw by the same (misguded imho) arguments "pure" 
output ranges ought to also be noncopyable).
Jun 24 2020
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Jun 24, 2020 at 12:29:13PM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:

 On 6/24/20 12:25 PM, Adam D. Ruppe wrote:

 I'm barely reading this thread, but could  live be useful with input
 ranges too?


 
 I'm not sure, but with a no-quarter-given approach to copying input
 ranges, simple tasks like "feed input from this file, or from stdin,
 and output to this other file or to stdout" mutate from trivial into
 little research projects (btw by the same (misguded imho) arguments
 "pure" output ranges ought to also be noncopyable).


I second this.  Whatever we decide to do about the input range / forward
range distinction, it should absolutely not cripple input ranges or make
them so different that you have to write two versions of the same
algorithm just to handle both cases.

For all of its flaws, the current range API does a beautiful job of
unifying the two (to the extent possible, modulo well-known bugs /
design limitations) so that, for the most part, you can ignore the
difference.  As long as .save isn't required, you could freely feed
either an input range or a forward range to any range algorithm, and
freely compose such algorithms, and it all Just Works(tm).

I propose that whatever new design we settle on for input ranges, it
should preserve this user-facing simplicity.  What we want is a design
where simple things are simple, and hard things are possible, not a
design where simple things are hard and hard things are nigh impossible.
To the extent possible, we should try to preserve symmetry between input
and forward ranges. (By "symmetry" I mean that as long as the equivalent
of .save isn't required, an input range and an output range ought to be
transparently interchangeable. A kind of Liskov Substitution like
principle applied to ranges.)


T

-- 
Latin's a dead language, as dead as can be; it killed off all the Romans, and
now it's killing me! -- Schoolboy
Jun 24 2020
prev sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Wednesday, 24 June 2020 at 16:25:31 UTC, Adam D. Ruppe wrote:

 I'm barely reading this thread, but could  live be useful with 
 input ranges too?


I don't think it can be, at least in its current form, but my 
reasoning behind making them non-copyable is pretty much the same 
as one behind  live and pointers: at any given moment, there can 
be only one owner of an input range's iteration state.
For some reason, Andrei equates that to not being able to read 
from the same source. ¯\_(ツ)_/¯
Jun 24 2020
parent Paul Backus <snarwin gmail.com> writes:
On Wednesday, 24 June 2020 at 20:05:27 UTC, Stanislav Blinov 
wrote:

 On Wednesday, 24 June 2020 at 16:25:31 UTC, Adam D. Ruppe wrote:

 I'm barely reading this thread, but could  live be useful with 
 input ranges too?


 I don't think it can be, at least in its current form, but my 
 reasoning behind making them non-copyable is pretty much the 
 same as one behind  live and pointers: at any given moment, 
 there can be only one owner of an input range's iteration state.
 For some reason, Andrei equates that to not being able to read 
 from the same source. ¯\_(ツ)_/¯


I think the missing piece here is that it's entirely possible for 
an input range to "borrow" its iteration state from somewhere 
else, rather than owning it. For example, a pointer to an input 
range is, itself, considered an input range, because of how the 
`.` operator does automatic dereferencing:

struct Example
{
     int i;

     bool empty() { return false; }
     int front() { return i; }
     void popFront() { ++i; }
}

static assert(isInputRange!(Example*));

Clearly there is no need for `Example*` to be non-copyable, even 
if we think `Example` ought to be.
Jun 25 2020
prev sibling next sibling parent Paul Backus <snarwin gmail.com> writes:
On Wednesday, 24 June 2020 at 16:17:58 UTC, Andrei Alexandrescu 
wrote:

 On 6/24/20 11:55 AM, Paul Backus wrote:

 To be clear: I agree with you *in principle* that pure input 
 ranges should be non-copyable.


 By what principle two input ranges should absolutely never use 
 the same data feed?


Point taken. An input range should either be non-copyable, or 
have reference semantics.
Jun 24 2020
prev sibling parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Wednesday, 24 June 2020 at 16:17:58 UTC, Andrei Alexandrescu 
wrote:

 On 6/24/20 11:55 AM, Paul Backus wrote:

 To be clear: I agree with you *in principle* that pure input 
 ranges should be non-copyable.


 By what principle two input ranges should absolutely never use 
 the same data feed?


By no principle. If a given data feed can be used in such manner 
- then used it shall be. By two distinct input ranges that keep 
their own state and don't breed references to said state.


 I'm not sure, but with a no-quarter-given approach to copying 
 input ranges, simple tasks      like "feed input from this 
 file, or from stdin, and output to this other file or to 
 stdout" mutate from trivial into little research projects


I don't see how. This and everything of the same simplicity stays 
exactly the same:

import std.stdio, std.array, std.algorithm;

void main()
{
     stdin
         .byLine
         .map!"a.length"
         .each!writeln;
}

Alleged issues materialize when you start using lvalues. Which 
I've already shown an example of a few posts back.

Does a CsvReader really need to allocate, especially if my data 
is numbers? The answer in current incarnation of ranges is "yes". 
Well then, I'd rather write one that doesn't. Thanks but no 
thanks, Phobos.


 (btw by the same (misguded imho) arguments "pure" output ranges 
 ought to also be noncopyable).


Perhaps.
Jun 24 2020
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/21/20 10:47 AM, Andrei Alexandrescu wrote:

 Was the better 
 alternative to make input ranges noncopyable? It only takes a couple of 
 hours of coding a bit of that up to see it's quite onerous.


To recap/put forth a few possible APIs for input ranges (that are not 
forward):

bool fetchNext(ref T target);

Spec: As long as the range is nonempty, assigns the next element to 
target and returns true. At end of range returns false without touching 
target.

Pros: works naturally with actual input ranges such as files and sockets.

Cons: does not work if T is qualified. Creates a copy of each element 
for forward ranges that exist in memory. That means essentially the API 
is impaired and most algorithms that work with input ranges would need 
to specialize for forward ranges, too. (FWIW this interface was first 
discussed before D had qualifiers). Composes so-so, consider 
implementations of filter (not bad) and map (not so nice).

T* fetchNext();

Spec: As long as the range is nonempty, returns a pointer to the next 
element in the range. At end of range returns null.

Pros: Simple and efficient for many ranges. Doesn't compose too well.

Cons: Issues with escape analysis and safety. Sometimes the pointer is 
scope, sometimes it's not depending on the range.

T fetchNext(ref bool done);
or
ref T fetchNext(ref bool done);

Spec: As long as the range is nonempty, sets done to false and returns 
the next element in the range. At end of range sets done to true and 
returns an arbitrary T.

Pros: works with qualified data.

Cons: inefficient or needs two versions depending on ref/value. 
Complicates life of clients.

Flag!"each" each(alias fun);

Spec: calls fun(x) for each element in the range x - an efficient 
generalization of opApply. If fun returns No.each, stops iteration (and 
also returns No.each). Otherwise, returns Yes.each. See 
https://dlang.org/library/std/algorithm/iteration/each.html.

Pros: works naturally and efficiently (assuming proper inlining) with 
many range types. Composes quite well, picture e.g. implementations of 
map and filter. Beautiful. Efficient implementation for forward ranges 
is immediate. Jives well with Oleg's famous take on iteration 
(http://okmij.org/ftp/papers/LL3-collections-talk.pdf).

Cons: ranges such as map, filter etc. would need to expose both each() 
(for input ranges) and the existing interface (for better than input 
ranges). A number of algorithms would need to be redone to take 
advantage of the new interface. The language integration is not as nice 
as with the (sadly inefficient) existing opApply.

At a point Sebastian Wilzbach and I discussed that several ranges should 
get an each() member function, but that never got finalized. Once each() 
is a member of prominent input range algos (can't hurt because people 
already can call the less efficient each() global function) and we 
accumulate experience with it, we can sanction it as a legit optional 
primitive for ranges.

One more thing before I forget - we should drop classes that are ranges. 
They add too much complication. The functionality would still be present 
by wrapping a polymorphic implementation in a struct.
Jun 21 2020
parent reply Jon Degenhardt <jond noreply.com> writes:
On Monday, 22 June 2020 at 02:52:57 UTC, Andrei Alexandrescu 
wrote:

 On 6/21/20 10:47 AM, Andrei Alexandrescu wrote:
 One more thing before I forget - we should drop classes that 
 are ranges. They add too much complication. The functionality 
 would still be present by wrapping a polymorphic implementation 
 in a struct.


I have used class-based ranges quite a bit to implement ranges 
with reference semantics. Classes are very convenient for this 
because the implementation is structurally very similar to a 
struct based implementation. Typically, replace 'struct' with 
'final class' and modify a helper function constructing the range 
to use 'new'. It's amazingly easy.

So - What aspects of class-based ranges is it valuable to drop? 
Polymorphism/inheritance? Reference semantics? Both?

I haven't found a need for polymorphism in these ranges, just 
reference semantics. I realize that the Phobos experience is the 
opposite, that reference ranges are rarely used. Could be that my 
design choices could be modified to eliminate reference ranges. 
Could also be that approaches needed in general libraries are not 
always necessary in code targeting specific applications. Not 
trying to make hard claims about this. Just pointing out where 
I've found value.

--Jon
Jun 22 2020
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Jun 22, 2020 at 10:54:08PM +0000, Jon Degenhardt via Digitalmars-d
wrote:

 On Monday, 22 June 2020 at 02:52:57 UTC, Andrei Alexandrescu wrote:

 On 6/21/20 10:47 AM, Andrei Alexandrescu wrote:
 One more thing before I forget - we should drop classes that are
 ranges.  They add too much complication. The functionality would
 still be present by wrapping a polymorphic implementation in a
 struct.


 
 I have used class-based ranges quite a bit to implement ranges with
 reference semantics. Classes are very convenient for this because the
 implementation is structurally very similar to a struct based
 implementation. Typically, replace 'struct' with 'final class' and
 modify a helper function constructing the range to use 'new'. It's
 amazingly easy.


[...]

Yeah, I rely on class ranges on occasion too, when runtime polymorphism
is required. They *are* cumbersome to use, though, I'll give you that.
If we're going to remove class-based ranges, then whatever replaces them
better have good support for runtime polymorphism, otherwise it's not
gonna fly.

This may be more common than people think. For example, sometimes I have
a range helper:

	auto myFunc(R)(R r, bool cond)
	{
		if (cond)
			return r.map!someFunc;
		else
			return r.filter!someFilter.map!someFunc;
	}

There is no way to make this compile (cond is only known at runtime),
because the return statements return diverse types, except to wrap it in
an InputRangeObject.

Though I only rarely need this, it's nonetheless critical functionality
because sometimes, you cannot know the exact range type until runtime,
and runtime polymorphism is unavoidable.  So this part of the current
range API cannot be just thrown out without a viable replacement.


T

-- 
Маленькие детки - маленькие бедки.
Jun 22 2020
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/22/20 11:12 PM, H. S. Teoh wrote:

 On Mon, Jun 22, 2020 at 10:54:08PM +0000, Jon Degenhardt via Digitalmars-d
wrote:

 On Monday, 22 June 2020 at 02:52:57 UTC, Andrei Alexandrescu wrote:

 On 6/21/20 10:47 AM, Andrei Alexandrescu wrote:
 One more thing before I forget - we should drop classes that are
 ranges.  They add too much complication. The functionality would
 still be present by wrapping a polymorphic implementation in a
 struct.


 I have used class-based ranges quite a bit to implement ranges with
 reference semantics. Classes are very convenient for this because the
 implementation is structurally very similar to a struct based
 implementation. Typically, replace 'struct' with 'final class' and
 modify a helper function constructing the range to use 'new'. It's
 amazingly easy.


 [...]
 
 Yeah, I rely on class ranges on occasion too, when runtime polymorphism
 is required. They *are* cumbersome to use, though, I'll give you that.
 If we're going to remove class-based ranges, then whatever replaces them
 better have good support for runtime polymorphism, otherwise it's not
 gonna fly.
 
 This may be more common than people think. For example, sometimes I have
 a range helper:
 
 	auto myFunc(R)(R r, bool cond)
 	{
 		if (cond)
 			return r.map!someFunc;
 		else
 			return r.filter!someFilter.map!someFunc;
 	}
 
 There is no way to make this compile (cond is only known at runtime),
 because the return statements return diverse types, except to wrap it in
 an InputRangeObject.


You don't need polymorphism for that (necessarily), just function 
pointers/delegates. I've also done this kind of stuff with iopipes using 
tagged unions:

https://github.com/schveiguy/httpiopipe/blob/a04d87de3aa3836c07d181263c399416ba005e7c/source/iopipe/http.d#L245-L252

Such a thing is also generalized in phobos: 
https://dlang.org/phobos/std_range.html#choose

And can probably be extended further (not sure why the args aren't lazy).

Your example rewritten:

auto myFunc(R)(R r, bool cond)
{
      return choose(cond, r.map!someFunc, r.filter!someFilter.map!someFunc);
}

-Steve
Jun 23 2020
prev sibling next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, June 22, 2020 9:12:02 PM MDT H. S. Teoh via Digitalmars-d wrote:

 On Mon, Jun 22, 2020 at 10:54:08PM +0000, Jon Degenhardt via Digitalmars-d 


wrote:

 On Monday, 22 June 2020 at 02:52:57 UTC, Andrei Alexandrescu wrote:

 On 6/21/20 10:47 AM, Andrei Alexandrescu wrote:
 One more thing before I forget - we should drop classes that are
 ranges.  They add too much complication. The functionality would
 still be present by wrapping a polymorphic implementation in a
 struct.


 I have used class-based ranges quite a bit to implement ranges with
 reference semantics. Classes are very convenient for this because the
 implementation is structurally very similar to a struct based
 implementation. Typically, replace 'struct' with 'final class' and
 modify a helper function constructing the range to use 'new'. It's
 amazingly easy.


 [...]

 Yeah, I rely on class ranges on occasion too, when runtime polymorphism
 is required. They *are* cumbersome to use, though, I'll give you that.
 If we're going to remove class-based ranges, then whatever replaces them
 better have good support for runtime polymorphism, otherwise it's not
 gonna fly.

 This may be more common than people think. For example, sometimes I have
 a range helper:

   auto myFunc(R)(R r, bool cond)
   {
       if (cond)
           return r.map!someFunc;
       else
           return r.filter!someFilter.map!someFunc;
   }

 There is no way to make this compile (cond is only known at runtime),
 because the return statements return diverse types, except to wrap it in
 an InputRangeObject.

 Though I only rarely need this, it's nonetheless critical functionality
 because sometimes, you cannot know the exact range type until runtime,
 and runtime polymorphism is unavoidable.  So this part of the current
 range API cannot be just thrown out without a viable replacement.


As far as forward ranges go, all that should be required is that the class
be wrapped by a struct where the copy constructor does the equivalent of
save. The class itself can then do the polymorphism like it would now. It's
just that the class won't be passed around directly. But because it's
wrapped in a struct, it then becomes possible to require that all forward
ranges have the same copying semantics (thus allowing us to ditch save) as
well as stuff like requiring that the init value be a valid, empty range
(which not only is not the case right now but cannot be the case so long as
we allow ranges which are classes).

That being said, it is almost always a mistake to make a range a class if it
can be avoided, because the allocations for save (or the copy constructor if
it replaces save) incur a huge performance hit relative to what it costs to
copy a range that's just a struct (e.g. the tests for dxml which use classes
for the range are _very_ slow in comparison to those that use dynamic arrays
or structs). So, even wrapped in a struct, it's not a great idea to use a
class for a forward range, but with problems like you describe here, AFAIK,
we really don't have a better solution. It's best to avoid the need for
runtime polymorphism with ranges, but it's not always possible. That being
said, it should work wrapped in a struct rather than having the class
exposed directly.

Now, as far as basic input ranges go, having them be classes isn't
necessarily a problem, since basic input ranges are essentially reference
types by their very nature (or at least, they can't be value types). If they
could have value semantics, they could be forward ranges. In fact, in some
ways, it would be nice to require that basic input ranges be classes, since
then it would be easy to introspect a basic input range vs a forward range,
and it would ensure that basic input ranges have full-on reference
semantics, but it also would result in heap allocations that are not
currently necessary - particularly in cases where it would currently work to
use a pseudo-reference type rather than full-on reference type. So, while it
would be very nice to be able to require that basic input ranges be classes,
I doubt that it's actually reasonable to do so. Either way, it shouldn't be
a problem to allow basic input ranges to be classes. It's just with forward
ranges that it's a problem, but struct wrappers should solve that problem.

- Jonathan M Davis
Jun 22 2020
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/22/20 6:54 PM, Jon Degenhardt wrote:

 On Monday, 22 June 2020 at 02:52:57 UTC, Andrei Alexandrescu wrote:

 On 6/21/20 10:47 AM, Andrei Alexandrescu wrote:
 One more thing before I forget - we should drop classes that are 
 ranges. They add too much complication. The functionality would still 
 be present by wrapping a polymorphic implementation in a struct.


 
 I have used class-based ranges quite a bit to implement ranges with 
 reference semantics. Classes are very convenient for this because the 
 implementation is structurally very similar to a struct based 
 implementation. Typically, replace 'struct' with 'final class' and 
 modify a helper function constructing the range to use 'new'. It's 
 amazingly easy.
 
 So - What aspects of class-based ranges is it valuable to drop? 
 Polymorphism/inheritance? Reference semantics? Both?
 
 I haven't found a need for polymorphism in these ranges, just reference 
 semantics. I realize that the Phobos experience is the opposite, that 
 reference ranges are rarely used. Could be that my design choices could 
 be modified to eliminate reference ranges. Could also be that approaches 
 needed in general libraries are not always necessary in code targeting 
 specific applications. Not trying to make hard claims about this. Just 
 pointing out where I've found value.


Problem is they're cumbersome to use - you need to remember to call 
save() in key places, and if you forget the compiler will be mum about 
it. If we do away with class ranges, save() can be eliminated in favor 
of the copy ctor.

I assume a wrapper that does define reference semantics where needed 
would not be difficult to write.
Jun 22 2020
next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, June 22, 2020 11:19:35 PM MDT Andrei Alexandrescu via Digitalmars-d 
wrote:

 On 6/22/20 6:54 PM, Jon Degenhardt wrote:

 On Monday, 22 June 2020 at 02:52:57 UTC, Andrei Alexandrescu wrote:

 On 6/21/20 10:47 AM, Andrei Alexandrescu wrote:
 One more thing before I forget - we should drop classes that are
 ranges. They add too much complication. The functionality would still
 be present by wrapping a polymorphic implementation in a struct.


 I have used class-based ranges quite a bit to implement ranges with
 reference semantics. Classes are very convenient for this because the
 implementation is structurally very similar to a struct based
 implementation. Typically, replace 'struct' with 'final class' and
 modify a helper function constructing the range to use 'new'. It's
 amazingly easy.

 So - What aspects of class-based ranges is it valuable to drop?
 Polymorphism/inheritance? Reference semantics? Both?

 I haven't found a need for polymorphism in these ranges, just reference
 semantics. I realize that the Phobos experience is the opposite, that
 reference ranges are rarely used. Could be that my design choices could
 be modified to eliminate reference ranges. Could also be that approaches
 needed in general libraries are not always necessary in code targeting
 specific applications. Not trying to make hard claims about this. Just
 pointing out where I've found value.


 Problem is they're cumbersome to use - you need to remember to call
 save() in key places, and if you forget the compiler will be mum about
 it. If we do away with class ranges, save() can be eliminated in favor
 of the copy ctor.

 I assume a wrapper that does define reference semantics where needed
 would not be difficult to write.


Except that if a forward range has reference semantics, then copying it is
not equivalent to calling save. If we get rid of save in favor of using copy
constructors, then we're requiring that all forward ranges have the same
copying semantics and that that be that copying a forward range results in a
copy that can be independently iterated over. Some internals could use
references to be sure, but we get rid of the ability to have forward ranges
which are actually reference types. They have to have the copying semantics
of value types with regards to at least their iteration state.

Now, one of the problems that we have with ranges as things stand is that
the semantics of copying a range are unspecified and can vary wildly
depending on the range's type, meaning that if a range is copied, you can't
ever use it again (at least in generic code), because the state of the
original is unknown after mutating the copy. It could be unchanged, be in
the same state as the copy, or even be in an invalid state due to the copy
mutating part of its state but not all of it. So, overall, having forward
ranges require that copying do what save now does would be a considerable
improvement with regards to cleaning up the copying semantics.

Of course, that still leaves the problem of basic input ranges, since they
can't have value semantics when copying (if they could, then they could be
forward ranges). Pretty much by definition, basic input ranges are either
reference types or pseudo-reference types (or the programmer was just lazy
and didn't implement save). So, unfortunately, generic code that operates on
both basic input ranges and forward ranges can't ever have guaranteed
copying semantics. I think that we should try to nail down the copying
semantics for ranges as much as we can though. Similarly, we should probably
place requirements on the semantics of assigning to a range. That would
almost certainly kill off RefRange, but IMHO, RefRange was a mistake anyway.

- Jonathan M Davis
Jun 22 2020
prev sibling parent Jon Degenhardt <jond noreply.com> writes:
On Tuesday, 23 June 2020 at 05:19:35 UTC, Andrei Alexandrescu 
wrote:

 On 6/22/20 6:54 PM, Jon Degenhardt wrote:

 On Monday, 22 June 2020 at 02:52:57 UTC, Andrei Alexandrescu 
 wrote:

 On 6/21/20 10:47 AM, Andrei Alexandrescu wrote:
 One more thing before I forget - we should drop classes that 
 are ranges. They add too much complication. The functionality 
 would still be present by wrapping a polymorphic 
 implementation in a struct.


 
 I have used class-based ranges quite a bit to implement ranges 
 with reference semantics. Classes are very convenient for this 
 because the implementation is structurally very similar to a 
 struct based implementation. Typically, replace 'struct' with 
 'final class' and modify a helper function constructing the 
 range to use 'new'. It's amazingly easy.


 Problem is they're cumbersome to use - you need to remember to 
 call save() in key places, and if you forget the compiler will 
 be mum about it. If we do away with class ranges, save() can be 
 eliminated in favor of the copy ctor.

 I assume a wrapper that does define reference semantics where 
 needed would not be difficult to write.


A couple thoughts. First, my uses for reference ranges have so 
far been on non-forward input ranges, so no 'save' function. No 
built-in member for copying either. I'm now wondering if there is 
some structural correlation between these choices and my 
application use cases, or if it is purely accidental.

Second - If I'm designing a range intended to have reference 
semantics, I'd expect it to be easier to build that choice into 
the main implementation of the range. If the implementation needs 
to be in two parts, a value-based, copyable range, and a separate 
wrapper to define reference semantics, then I'd expect it to be 
more complicated. Even if the wrapper is found in the standard 
library.

Simple things like decisions on how to organize unit tests. Are 
there two sets of unit tests, one for the value-based layer, and 
a second for wrapped layer? (I.e. If the value based layer can 
never be directly instantiated, should it have standalone unit 
tests?). And slightly harder things, like can the value based 
implementation layer assume it is never instantiated as a 
standalone value-based range? It sounds like more design 
decisions to achieve the goal.
Jun 22 2020
prev sibling next sibling parent reply Avrina <avrina12309412342 gmail.com> writes:
On Saturday, 20 June 2020 at 14:51:32 UTC, Andrei Alexandrescu 
wrote:

 On 6/20/20 5:49 AM, Stanislav Blinov wrote:

 On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:

 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu 
 via Digitalmars-d wrote: [...]

 One good goal for std.v2020 would be to forego autodecoding 
 throughout.


 [...]

 Another could be to fix up the range API -- i.e, reconsider 
 the ugliness that is .save, now that D has copy ctors.


 
 How do you see that? Fundamentally, what have copy ctors 
 changed in this regard?


 I've been thinking of this for a while. The fact that input 
 ranges must buffer one element (i.e. make front() callable 
 several times) has been a gauche choice.

 Input ranges should have only one API:

 bool fetchNext(T& target);

 Fill the user-provided target with the next element and return 
 true. At the end of the range, return false and leave the 
 target alone.

 I'd considered this design back in the day but I was annoyed 
 that it required a copy for ranges that in fact are better than 
 input ranges (e.g. arrays etc).

 It's still a disadvantage, but over the years I realized there 
 are not that many algorithms that only work on input ranges so 
 we could make-do. Of those for which efficiency is a concern, 
 they could easily be specialized for forward ranges.


Most of the algorithms in Phobos don't call front() multiple 
times and they make a copy. It is really annoying as this 
prevents using non copyable types or those that need to be moved, 
eg similar to unique_ptr. What you are suggesting doesn't sound 
all that better, it just cements what is already common place in 
Phobos into the API. It's usually not algorithms that are 
concerned with efficiency, but rather the type. If you implement 
an algorithm that you think doesn't need to be efficient but it 
does need to be for the type I am using it with, I don't think 
that'd any better than the situation we already have.
Jun 20 2020
next sibling parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 20 June 2020 at 17:02:56 UTC, Avrina wrote:


 Most of the algorithms in Phobos don't call front() multiple 
 times and they make a copy. It is really annoying as this 
 prevents using non copyable types or those that need to be 
 moved, eg similar to unique_ptr. What you are suggesting 
 doesn't sound all that better, it just cements what is already 
 common place in Phobos into the API.


On the contrary. Non-copyable types are *the* target for a 
hypothetical fetchNext(ref T target). Which would be  system, and 
you'd be calling it with an uninitialized value.
Although a one-pass consuming range could also be built with the 
`moveFront` interface.

That said, some algorithms do indeed require an improvement 
(toward moving instead of copying). `sort` just recently got such 
improvement courtesy of MoonlightSentinel.
Jun 20 2020
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/20 1:02 PM, Avrina wrote:

 Most of the algorithms in Phobos don't call front() multiple times and 
 they make a copy.


Interesting. They should be easy to fix then. Got a few examples?
Jun 20 2020
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, June 20, 2020 6:07:29 PM MDT Andrei Alexandrescu via Digitalmars-
d wrote:

 On 6/20/20 1:02 PM, Avrina wrote:

 Most of the algorithms in Phobos don't call front() multiple times and
 they make a copy.


 Interesting. They should be easy to fix then. Got a few examples?


Part of the problem is that sometimes, you very much want to be calling
front only once and storing the result, because front is calculated every
time that it's called (e.g. map calculates front on every call), whereas
other times, you want to call front multiple times so that you can avoid
copying the return value. To make matters worse, in the case of something
like map, while you might get equivalent values on each call to front, you
won't necessarily get exactly the same value with each call to front (e.g.
if map's predicate calls new). So, there's a strong argument to be made that
it's more correct and likely more efficient to call front only once even if
that means copying the return value. However, AFAIK, there isn't really
consensus on what the correct approach is for generic code, and I don't know
what the more common approach is in Phobos. I suppose that code could do
introspection on front to see whether it returned by ref such that it called
front multiple times if it returned by ref and avoided calling it multiple
times if it didn't, but I rather doubt that much of anyone is going to do
something like that in practice.

Obviously, if non-copyable types were to be supported, then it needs to be
the norm to avoid copying the return value of front, but non-copyable types
are a bit of a disaster with ranges anyway, since it's so trivial for them
to result in the range itself being non-copyable (at which point, it doesn't
even work with foreach), and it's very common for algorithms to assume that
a range's elements are copyable. It also risks having to have code test for
copyability rather than assuming that it's possible, which will add yet
another stray ability that range-based code will have to test for.
Personally, I don't think that supporting non-copyable types is worth it and
that isInputRange should just ban them, but obviously, some folks want
non-copyable types to be supported. However, isForwardRange does currently
ban them (though I don't know if it's on purpose on just a happy accident).
So, there's that.

- Jonathan M Davis
Jun 22 2020
prev sibling next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Saturday, 20 June 2020 at 14:51:32 UTC, Andrei Alexandrescu 
wrote:

 [snip]

 I've been thinking of this for a while. The fact that input 
 ranges must buffer one element (i.e. make front() callable 
 several times) has been a gauche choice.

 Input ranges should have only one API:

 bool fetchNext(T& target);

 Fill the user-provided target with the next element and return 
 true. At the end of the range, return false and leave the 
 target alone.

 I'd considered this design back in the day but I was annoyed 
 that it required a copy for ranges that in fact are better than 
 input ranges (e.g. arrays etc).

 It's still a disadvantage, but over the years I realized there 
 are not that many algorithms that only work on input ranges so 
 we could make-do. Of those for which efficiency is a concern, 
 they could easily be specialized for forward ranges.


That similar to what Steven Schveighoffer did in dcollections 
with cursors.

https://github.com/schveiguy/dcollections/blob/master/concepts.txt
Jun 20 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/20/20 2:04 PM, jmh530 wrote:

 On Saturday, 20 June 2020 at 14:51:32 UTC, Andrei Alexandrescu wrote:

 [snip]

 I've been thinking of this for a while. The fact that input ranges 
 must buffer one element (i.e. make front() callable several times) has 
 been a gauche choice.

 Input ranges should have only one API:

 bool fetchNext(T& target);

 Fill the user-provided target with the next element and return true. 
 At the end of the range, return false and leave the target alone.

 I'd considered this design back in the day but I was annoyed that it 
 required a copy for ranges that in fact are better than input ranges 
 (e.g. arrays etc).

 It's still a disadvantage, but over the years I realized there are not 
 that many algorithms that only work on input ranges so we could 
 make-do. Of those for which efficiency is a concern, they could easily 
 be specialized for forward ranges.


 
 That similar to what Steven Schveighoffer did in dcollections with cursors.
 
 https://github.com/schveiguy/dcollections/blob/master/concepts.txt


Sorry, this is not the same.

Cursors are simply a way to refer to exactly one element, instead of 
using a begin and end element like a traditional container range would.

The advantage is that removing the "end" element (that has nothing to do 
with the range) won't invalidate the cursor.

What I wanted was the functionality of C++ iterators without the danger 
that comes with actually iterating them.

-Steve
Jun 20 2020
parent jmh530 <john.michael.hall gmail.com> writes:
On Saturday, 20 June 2020 at 18:33:46 UTC, Steven Schveighoffer 
wrote:

 [snip]

 Sorry, this is not the same.

 Cursors are simply a way to refer to exactly one element, 
 instead of using a begin and end element like a traditional 
 container range would.

 The advantage is that removing the "end" element (that has 
 nothing to do with the range) won't invalidate the cursor.

 What I wanted was the functionality of C++ iterators without 
 the danger that comes with actually iterating them.

 -Steve


Andrei's proposal was for a function that returns true and fills 
the referred to variable and returns false otherwise. Cursors are 
obviously different, but they have a similarity in that a simple 
cursor, like below, has true/false behavior by storing it in 
empty.

struct Cursor(T) {
     T* ptr;
     bool empty = false;
}

That aspect of it was what I was thinking of. So for instance, if 
you have a struct that has a Cursor as a member, then you could 
do something like below.
bool fetchNext(ref T target) {
     cursor.popFront; //need to define popFront
     target = cursor.ptr;
     return cursor.empty;
}
Jun 20 2020
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 20.06.20 16:51, Andrei Alexandrescu wrote:

 
 Input ranges should have only one API:
 
 bool fetchNext(T& target);
 
 Fill the user-provided target with the next element and return true. At 
 the end of the range, return false and leave the target alone.


What if the caller does not know how to construct a T?

Nullable!T fetchNext();

(I get that D's support for algebraic data types is subpar, but maybe 
that is something to look into.)
Jun 21 2020
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/21/20 6:06 PM, Timon Gehr wrote:

 On 20.06.20 16:51, Andrei Alexandrescu wrote:

 Input ranges should have only one API:

 bool fetchNext(T& target);

 Fill the user-provided target with the next element and return true. 
 At the end of the range, return false and leave the target alone.


 
 What if the caller does not know how to construct a T?
 
 Nullable!T fetchNext();
 
 (I get that D's support for algebraic data types is subpar, but maybe 
 that is something to look into.)


That also creates a copy for each element, which may be disadvantageous 
for certain range types. (Also does not give access to references to the 
original elements, for you can't say Nullable!(ref T).)

Making an API that is simple and works with in-memory elements as well 
as values synthesized on the fly is quite a challenge.
Jun 21 2020
prev sibling next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:

 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu 
 via Digitalmars-d wrote: [...]

 One good goal for std.v2020 would be to forego autodecoding 
 throughout.


 [...]

 Another could be to fix up the range API -- i.e, reconsider the 
 ugliness that is .save, now that D has copy ctors.


 T


Also, switch from `void popFront()` to `typeof(this) rest`, so 
that we can have `const` and `immutable` ranges.
Jun 20 2020
next sibling parent Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Saturday, 20 June 2020 at 10:43:41 UTC, Paul Backus wrote:

 On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:

 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu 
 via Digitalmars-d wrote: [...]

 One good goal for std.v2020 would be to forego autodecoding 
 throughout.


 [...]

 Another could be to fix up the range API -- i.e, reconsider 
 the ugliness that is .save, now that D has copy ctors.


 T


 Also, switch from `void popFront()` to `typeof(this) rest`, so 
 that we can have `const` and `immutable` ranges.


Yes!
Jun 20 2020
prev sibling next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Saturday, 20 June 2020 at 10:43:41 UTC, Paul Backus wrote:

 [snip]

 Also, switch from `void popFront()` to `typeof(this) rest`, so 
 that we can have `const` and `immutable` ranges.


Would typeof(this) work where popFront is currently a 
free-standing function, like with built-in dynamic arrays?
Jun 20 2020
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Saturday, 20 June 2020 at 12:05:18 UTC, jmh530 wrote:

 On Saturday, 20 June 2020 at 10:43:41 UTC, Paul Backus wrote:

 [snip]

 Also, switch from `void popFront()` to `typeof(this) rest`, so 
 that we can have `const` and `immutable` ranges.


 Would typeof(this) work where popFront is currently a 
 free-standing function, like with built-in dynamic arrays?


typeof(this) is only possible for member function based 
implementations. Otherwise the implementation for arrays would be:

inout(T)[] rest(T)(input(T)[] array) { return array[1 .. $]; }

https://run.dlang.io/gist/run-dlang/7a4e5872ee26c083b916303fe36ac43b?compiler=dmd
Jun 20 2020
parent jmh530 <john.michael.hall gmail.com> writes:
On Saturday, 20 June 2020 at 12:26:55 UTC, Petar Kirov 
[ZombineDev] wrote:

 [snip]

 typeof(this) is only possible for member function based 
 implementations. Otherwise the implementation for arrays would 
 be:

 inout(T)[] rest(T)(input(T)[] array) { return array[1 .. $]; }

 https://run.dlang.io/gist/run-dlang/7a4e5872ee26c083b916303fe36ac43b?compiler=dmd


Gotcha, thanks.
Jun 20 2020
prev sibling next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 20 June 2020 at 10:43:41 UTC, Paul Backus wrote:


 Also, switch from `void popFront()` to `typeof(this) rest`, so 
 that we can have `const` and `immutable` ranges.


*Switch* is probably too restrictive. For a given range a 
`popFront` may be more efficient than a `range = range.rest`. 
Making `rest` a valid range primitive though, and using where 
appropriate - that'd be awesome.
Jun 20 2020
next sibling parent Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Saturday, 20 June 2020 at 12:42:01 UTC, Stanislav Blinov wrote:

 On Saturday, 20 June 2020 at 10:43:41 UTC, Paul Backus wrote:


 Also, switch from `void popFront()` to `typeof(this) rest`, so 
 that we can have `const` and `immutable` ranges.


 *Switch* is probably too restrictive. For a given range a 
 `popFront` may be more efficient than a `range = range.rest`. 
 Making `rest` a valid range primitive though, and using where 
 appropriate - that'd be awesome.


Yes, that may be a valid concern, I have thought about this as 
well.

The main issue is for some kinds of iterables, like alias 
sequences, we have no choice but to use an immutable-friendly API 
like rest/dropOne or concat (not append). A couple of years ago I 
was able to write proof-of-concept library that had accumulate, 
map, and filter implementations that worked with both ranges and 
template parameter sequences, based on immutable-friendly 
functions like this

See:

https://github.com/PetarKirov/rxd/blob/v0.0.3/source/rxd/xf/xform.d

Some of the technics that I had developed allowed me to implement 
UFCS-enabled alternatives to std.meta that accept a polymorphic 
lambda parameter for mapping/filtering/etc.:

https://gist.github.com/PetarKirov/a808c94857de84858accfb094c19bf77
Jun 20 2020
prev sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Saturday, 20 June 2020 at 12:42:01 UTC, Stanislav Blinov wrote:

 On Saturday, 20 June 2020 at 10:43:41 UTC, Paul Backus wrote:


 Also, switch from `void popFront()` to `typeof(this) rest`, so 
 that we can have `const` and `immutable` ranges.


 *Switch* is probably too restrictive. For a given range a 
 `popFront` may be more efficient than a `range = range.rest`. 
 Making `rest` a valid range primitive though, and using where 
 appropriate - that'd be awesome.


For non-forward ranges, there's no promise that the original 
range (or any copies of it) will remain valid after calling 
`rest`, so you can always implement `rest` like this:

     auto rest() {
         this.popFront;
         return this;
     }
Jun 20 2020
next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 20 June 2020 at 13:07:41 UTC, Paul Backus wrote:

 On Saturday, 20 June 2020 at 12:42:01 UTC, Stanislav Blinov 
 wrote:

 On Saturday, 20 June 2020 at 10:43:41 UTC, Paul Backus wrote:


 Also, switch from `void popFront()` to `typeof(this) rest`, 
 so that we can have `const` and `immutable` ranges.


 *Switch* is probably too restrictive. For a given range a 
 `popFront` may be more efficient than a `range = range.rest`. 
 Making `rest` a valid range primitive though, and using where 
 appropriate - that'd be awesome.


 For non-forward ranges, there's no promise that the original 
 range (or any copies of it) will remain valid after calling 
 `rest`, so you can always implement `rest` like this:

     auto rest() {
         this.popFront;
         return this;
     }


For sure. I'm just pointing out that *switching* to `rest` isn't 
a likely way to go. I.e. that would've implied dropping the use 
of popFront altogether :) *Adopting* `rest` - yes, +100%.
Jun 20 2020
parent Paul Backus <snarwin gmail.com> writes:
On Saturday, 20 June 2020 at 13:29:43 UTC, Stanislav Blinov wrote:

 On Saturday, 20 June 2020 at 13:07:41 UTC, Paul Backus wrote:

 On Saturday, 20 June 2020 at 12:42:01 UTC, Stanislav Blinov 
 wrote:

 On Saturday, 20 June 2020 at 10:43:41 UTC, Paul Backus wrote:


 Also, switch from `void popFront()` to `typeof(this) rest`, 
 so that we can have `const` and `immutable` ranges.


 *Switch* is probably too restrictive. For a given range a 
 `popFront` may be more efficient than a `range = range.rest`. 
 Making `rest` a valid range primitive though, and using where 
 appropriate - that'd be awesome.


 For non-forward ranges, there's no promise that the original 
 range (or any copies of it) will remain valid after calling 
 `rest`, so you can always implement `rest` like this:

     auto rest() {
         this.popFront;
         return this;
     }


 For sure. I'm just pointing out that *switching* to `rest` 
 isn't a likely way to go. I.e. that would've implied dropping 
 the use of popFront altogether :) *Adopting* `rest` - yes, 
 +100%.


By "switching", I just mean that `std.v2.range.isInputRange` 
would only check for `rest`, not `popFront`, and that new code 
would be free to only implement `rest`. Naturally, we would also 
provide free-function versions of `rest` and `popFront` for 
compatibility between the two APIs.
Jun 20 2020
prev sibling parent reply Johannes Loher <johannes.loher fg4f.de> writes:
On Saturday, 20 June 2020 at 13:07:41 UTC, Paul Backus wrote:

 On Saturday, 20 June 2020 at 12:42:01 UTC, Stanislav Blinov 
 wrote:

 On Saturday, 20 June 2020 at 10:43:41 UTC, Paul Backus wrote:


 Also, switch from `void popFront()` to `typeof(this) rest`, 
 so that we can have `const` and `immutable` ranges.


 *Switch* is probably too restrictive. For a given range a 
 `popFront` may be more efficient than a `range = range.rest`. 
 Making `rest` a valid range primitive though, and using where 
 appropriate - that'd be awesome.


 For non-forward ranges, there's no promise that the original 
 range (or any copies of it) will remain valid after calling 
 `rest`, so you can always implement `rest` like this:

     auto rest() {
         this.popFront;
         return this;
     }


Can’t make it const then... also some ranges probably cannot bee 
const by design, e.g. byLine etc.
Jun 20 2020
next sibling parent Paul Backus <snarwin gmail.com> writes:
On Saturday, 20 June 2020 at 18:34:37 UTC, Johannes Loher wrote:

 On Saturday, 20 June 2020 at 13:07:41 UTC, Paul Backus wrote:

 For non-forward ranges, there's no promise that the original 
 range (or any copies of it) will remain valid after calling 
 `rest`, so you can always implement `rest` like this:

     auto rest() {
         this.popFront;
         return this;
     }


 Can’t make it const then... also some ranges probably cannot 
 bee const by design, e.g. byLine etc.


Yes, ranges with popFront can't be const. The difference is that, 
with `rest`/`tail` as the default, *some* ranges that are 
currently forced to use popFront unnecessarily can be made 
const-compatible.

Ranges that continue to use popFront, for whatever reason 
(performance, backwards compatibility, etc.) will remain 
mutable-only.
Jun 20 2020
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/20 2:34 PM, Johannes Loher wrote:

 On Saturday, 20 June 2020 at 13:07:41 UTC, Paul Backus wrote:

 On Saturday, 20 June 2020 at 12:42:01 UTC, Stanislav Blinov wrote:

 On Saturday, 20 June 2020 at 10:43:41 UTC, Paul Backus wrote:


 Also, switch from `void popFront()` to `typeof(this) rest`, so that 
 we can have `const` and `immutable` ranges.


 *Switch* is probably too restrictive. For a given range a `popFront` 
 may be more efficient than a `range = range.rest`. Making `rest` a 
 valid range primitive though, and using where appropriate - that'd be 
 awesome.


 For non-forward ranges, there's no promise that the original range (or 
 any copies of it) will remain valid after calling `rest`, so you can 
 always implement `rest` like this:

     auto rest() {
         this.popFront;
         return this;
     }


 
 Can’t make it const then... also some ranges probably cannot bee const 
 by design, e.g. byLine etc.


That function would be a fallback. Ranges that are const would implement 
it as a primitive not expressible with popFront.

Consider:

immutable List(T) {
     private T payload;
     private List next;
     public List tail() { return next; }
     ...
}

You can't implement popFront for this structure.
Jun 20 2020
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/20 6:43 AM, Paul Backus wrote:

 On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:

 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu via 
 Digitalmars-d wrote: [...]

 One good goal for std.v2020 would be to forego autodecoding throughout.


 [...]

 Another could be to fix up the range API -- i.e, reconsider the 
 ugliness that is .save, now that D has copy ctors.


 T


 
 Also, switch from `void popFront()` to `typeof(this) rest`, so that we 
 can have `const` and `immutable` ranges.


Yah, tail() would be necessary too. popFront needs to stay because it 
doesn't copy the range.
Jun 20 2020
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/20/20 6:43 AM, Paul Backus wrote:

 On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:

 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu via 
 Digitalmars-d wrote: [...]

 One good goal for std.v2020 would be to forego autodecoding throughout.


 [...]

 Another could be to fix up the range API -- i.e, reconsider the 
 ugliness that is .save, now that D has copy ctors.


 T


 
 Also, switch from `void popFront()` to `typeof(this) rest`, so that we 
 can have `const` and `immutable` ranges.


How does that work? You'd have to use recursion I guess? ugly. Why do we 
need ranges for something like this?

-Steve
Jun 20 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/20 3:03 PM, Steven Schveighoffer wrote:

 On 6/20/20 6:43 AM, Paul Backus wrote:

 On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:

 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu via 
 Digitalmars-d wrote: [...]

 One good goal for std.v2020 would be to forego autodecoding throughout.


 [...]

 Another could be to fix up the range API -- i.e, reconsider the 
 ugliness that is .save, now that D has copy ctors.


 T


 Also, switch from `void popFront()` to `typeof(this) rest`, so that we 
 can have `const` and `immutable` ranges.


 
 How does that work? You'd have to use recursion I guess? ugly. Why do we 
 need ranges for something like this?


Think Hakell lists. They can implement tail easily (just return the next 
pointer) but can't define popFront.
Jun 20 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/20/20 8:12 PM, Andrei Alexandrescu wrote:

 On 6/20/20 3:03 PM, Steven Schveighoffer wrote:

 On 6/20/20 6:43 AM, Paul Backus wrote:

 On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:

 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu via 
 Digitalmars-d wrote: [...]

 One good goal for std.v2020 would be to forego autodecoding 
 throughout.


 [...]

 Another could be to fix up the range API -- i.e, reconsider the 
 ugliness that is .save, now that D has copy ctors.


 Also, switch from `void popFront()` to `typeof(this) rest`, so that 
 we can have `const` and `immutable` ranges.


 How does that work? You'd have to use recursion I guess? ugly. Why do 
 we need ranges for something like this?


 
 Think Hakell lists. They can implement tail easily (just return the next 
 pointer) but can't define popFront.
 


My question wasn't about how such a thing could be implemented, but how 
it works with const ranges.

foreach(x; someConstRange) I think wouldn't be possible. I think you'd 
have to recurse:

void process(const Range r)
{
    subProcess(r.front);
    process(r.rest);
}

The point is to question the statement "so that we can have `const` and 
`immutable` ranges".

Sure, we could implement recursive versions of find, etc. I don't know 
if that's worth it.

-Steve
Jun 22 2020
next sibling parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Monday, 22 June 2020 at 12:15:39 UTC, Steven Schveighoffer 
wrote:

 On 6/20/20 8:12 PM, Andrei Alexandrescu wrote:

 On 6/20/20 3:03 PM, Steven Schveighoffer wrote:

 On 6/20/20 6:43 AM, Paul Backus wrote:

 On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:

 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei 
 Alexandrescu via Digitalmars-d wrote: [...]

 One good goal for std.v2020 would be to forego 
 autodecoding throughout.


 [...]

 Another could be to fix up the range API -- i.e, reconsider 
 the ugliness that is .save, now that D has copy ctors.


 Also, switch from `void popFront()` to `typeof(this) rest`, 
 so that we can have `const` and `immutable` ranges.


 How does that work? You'd have to use recursion I guess? 
 ugly. Why do we need ranges for something like this?


 
 Think Hakell lists. They can implement tail easily (just 
 return the next pointer) but can't define popFront.
 


 My question wasn't about how such a thing could be implemented, 
 but how it works with const ranges.

 foreach(x; someConstRange) I think wouldn't be possible. I 
 think you'd have to recurse:

 void process(const Range r)
 {
    subProcess(r.front);
    process(r.rest);
 }


I think maybe what is needed is a notion of "head-mutable"?

void process(const headmut Range r)
{
   while (!r.empty) {
     // subProcess still has to take an immmutable r. this does 
not expose mutable state.
     // headmut converts to immutable on pass-by-value because it 
makes a copy.
     subProcess(r.front);
     r = r.rest; // can be assigned because headmut, meaning 
references are still immutable
   }
}

I don't know how this would interact with dip1000. Maybe headmut 
could convert as immutable but expression scoped, which would 
come down to the same thing.

Something like this is also probably the right backing type to 
use for many containers, like Nullable.
Jun 22 2020
parent reply Seb <seb wilzba.ch> writes:
On Monday, 22 June 2020 at 12:28:06 UTC, FeepingCreature wrote:

 I think maybe what is needed is a notion of "head-mutable"?


It's called Final and already in Phobos:

https://dlang.org/phobos/std_experimental_typecons.html#.Final

Though it comes with a few bugs like e.g. [1] or [2].

[1] https://issues.dlang.org/show_bug.cgi?id=17272
[2] https://github.com/dlang/phobos/pull/6596
Jun 22 2020
parent reply Seb <seb wilzba.ch> writes:
On Monday, 22 June 2020 at 15:29:25 UTC, Seb wrote:

 On Monday, 22 June 2020 at 12:28:06 UTC, FeepingCreature wrote:

 I think maybe what is needed is a notion of "head-mutable"?


 It's called Final and already in Phobos:

 https://dlang.org/phobos/std_experimental_typecons.html#.Final

 Though it comes with a few bugs like e.g. [1] or [2].

 [1] https://issues.dlang.org/show_bug.cgi?id=17272
 [2] https://github.com/dlang/phobos/pull/6596


Sorry. I read to quick. Final is head-const and tail-mutable.
Rebindable is the Phobos solution to head-mutable / tail-const:

https://dlang.org/phobos/std_typecons.html#Rebindable

The biggest gotcha of Rebindable is that it doesn't work with 
structs [1] which vastly limits its usability.

[1] https://github.com/dlang/phobos/pull/6136
Jun 22 2020
parent FeepingCreature <feepingcreature gmail.com> writes:
On Monday, 22 June 2020 at 15:34:36 UTC, Seb wrote:

 The biggest gotcha of Rebindable is that it doesn't work with 
 structs [1] which vastly limits its usability.

 [1] https://github.com/dlang/phobos/pull/6136


Yes, effectively what I'm looking for is "Rebindable for 
immutable structs." I already have this in Nullable, but it's an 
atrocious hack because I basically have to hit the compiler on 
the head until it forgets what const is. I want some way to say 
"yes, this type is immutable, but it's only immutable over its 
lifetime, and *I* control its lifetime and *I* say when it ends - 
and when it ends, I'm free to put other data there, such as the 
return value of rest()."

This is also vital for finally allowing immutable types in 
(key-)mutable hashmaps.
Jun 23 2020
prev sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 22 June 2020 at 12:15:39 UTC, Steven Schveighoffer 
wrote:

 My question wasn't about how such a thing could be implemented, 
 but how it works with const ranges.

 foreach(x; someConstRange) I think wouldn't be possible. I 
 think you'd have to recurse:

 void process(const Range r)
 {
    subProcess(r.front);
    process(r.rest);
 }

 The point is to question the statement "so that we can have 
 `const` and `immutable` ranges".

 Sure, we could implement recursive versions of find, etc. I 
 don't know if that's worth it.

 -Steve


Well, currently, range algorithms can't work with const ranges 
*at all*, recursively or iteratively. So from a user perspective, 
this would be a strict improvement on the status quo.
Jun 22 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/22/20 10:20 AM, Paul Backus wrote:

 On Monday, 22 June 2020 at 12:15:39 UTC, Steven Schveighoffer wrote:

 My question wasn't about how such a thing could be implemented, but 
 how it works with const ranges.

 foreach(x; someConstRange) I think wouldn't be possible. I think you'd 
 have to recurse:

 void process(const Range r)
 {
    subProcess(r.front);
    process(r.rest);
 }

 The point is to question the statement "so that we can have `const` 
 and `immutable` ranges".

 Sure, we could implement recursive versions of find, etc. I don't know 
 if that's worth it.


 
 Well, currently, range algorithms can't work with const ranges *at all*, 
 recursively or iteratively. So from a user perspective, this would be a 
 strict improvement on the status quo.


Algorithms can work with const ranges -- as long as the range is an array:

const int[] arr = [1, 2, 3, 4, 5];

auto x = arr.find(3);

assert(x == [3, 4, 5]);

I think the better option is to focus on making it possible to duplicate 
this possibility for generic ranges rather than implement a new and 
awkward API.

FeepingCreature is right, we should try and create a head mutable 
mechanism. I have envisioned it in the past as a tail-modifier mechanism 
(e.g. tail-const).

-Steve
Jun 22 2020
next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 22 June 2020 at 14:33:53 UTC, Steven Schveighoffer 
wrote:

 On 6/22/20 10:20 AM, Paul Backus wrote:

 
 Well, currently, range algorithms can't work with const ranges 
 *at all*, recursively or iteratively. So from a user 
 perspective, this would be a strict improvement on the status 
 quo.


 Algorithms can work with const ranges -- as long as the range 
 is an array:

 const int[] arr = [1, 2, 3, 4, 5];

 auto x = arr.find(3);

 assert(x == [3, 4, 5]);

 I think the better option is to focus on making it possible to 
 duplicate this possibility for generic ranges rather than 
 implement a new and awkward API.

 FeepingCreature is right, we should try and create a head 
 mutable mechanism. I have envisioned it in the past as a 
 tail-modifier mechanism (e.g. tail-const).

 -Steve


This isn't really "algorithms working with const ranges"; rather, 
it's "const(T[]) implicitly converts to const(T)[]". The 
algorithm itself never sees a const range.

I suppose we could have the same thing for user-defined types if 
we added ` implicit opCast`, or a more restrictive version like 
`opHeadMutable`, but the whole thing seems backwards to me. 
const(T[]) converting to const(T)[] is a special case in the 
language introduced to compensate for the shortcomings 
`popFront`. Rather than doubling down on it, why not fix the 
issue at its source?
Jun 22 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/22/20 10:55 AM, Paul Backus wrote:

 On Monday, 22 June 2020 at 14:33:53 UTC, Steven Schveighoffer wrote:

 On 6/22/20 10:20 AM, Paul Backus wrote:

 Well, currently, range algorithms can't work with const ranges *at 
 all*, recursively or iteratively. So from a user perspective, this 
 would be a strict improvement on the status quo.


 Algorithms can work with const ranges -- as long as the range is an 
 array:

 const int[] arr = [1, 2, 3, 4, 5];

 auto x = arr.find(3);

 assert(x == [3, 4, 5]);

 I think the better option is to focus on making it possible to 
 duplicate this possibility for generic ranges rather than implement a 
 new and awkward API.

 FeepingCreature is right, we should try and create a head mutable 
 mechanism. I have envisioned it in the past as a tail-modifier 
 mechanism (e.g. tail-const).


 
 This isn't really "algorithms working with const ranges"; rather, it's 
 "const(T[]) implicitly converts to const(T)[]". The algorithm itself 
 never sees a const range.


I don't see a difference. When you copy a range as a parameter, the head 
is a different piece of memory. This is why it works. Why is it 
important how it works?


 I suppose we could have the same thing for user-defined types if we 
 added ` implicit opCast`, or a more restrictive version like 
 `opHeadMutable`, but the whole thing seems backwards to me. const(T[]) 
 converting to const(T)[] is a special case in the language introduced to 
 compensate for the shortcomings `popFront`. Rather than doubling down on 
 it, why not fix the issue at its source?


That isn't why tail-const was "introduced". And I would consider 
enabling this feature for custom types fixing it at the source.

Why does it seem backwards to you?

Perhaps you are thinking of how templates automatically strip the head 
const? That I don't necessarily agree with either, but there isn't a 
good way to say "take this parameter as head-mutable" in generic code, 
which is why *that* was added. If we did have a mechanism to say that 
(such as FeepingCreture's `headmut` example), then we wouldn't need that 
special treatment.

-Steve
Jun 22 2020
parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 22 June 2020 at 15:47:51 UTC, Steven Schveighoffer 
wrote:

 On 6/22/20 10:55 AM, Paul Backus wrote:

 
 This isn't really "algorithms working with const ranges"; 
 rather, it's "const(T[]) implicitly converts to const(T)[]". 
 The algorithm itself never sees a const range.


 I don't see a difference. When you copy a range as a parameter, 
 the head is a different piece of memory. This is why it works. 
 Why is it important how it works?


It's important how it works because it *doesn't* work, for ranges 
in general. It only works for slices.

Making it work in general requires either subverting the type 
system (like Rebindable does) or adding more special cases to the 
language.


 Perhaps you are thinking of how templates automatically strip 
 the head const? That I don't necessarily agree with either, but 
 there isn't a good way to say "take this parameter as 
 head-mutable" in generic code, which is why *that* was added. 
 If we did have a mechanism to say that (such as 
 FeepingCreture's `headmut` example), then we wouldn't need that 
 special treatment.


The question is, why do you need a mechanism to say that in the 
first place? Why not just have generic code take the argument as 
its actual type?

Answer: because popFront requires a mutable object.

`headmut` is a solution to a problem that doesn't need to exist 
in the first place. Get rid of popFront, and you get rid of the 
problem.
Jun 22 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 6/22/20 12:14 PM, Paul Backus wrote:

 Making it work in general requires either subverting the type system 
 (like Rebindable does) or adding more special cases to the language.


The principled approach is to generalize the trick currently used only 
for arrays. Consider the lowering:

"Whenever something is passed to a function, .opOnCall is appended."

Then:

auto ref T opOnCall(T)(auto ref T x) { return x; }
const(T)[] opOnCall(T)(const(T[]) x) { return x; }
immutable(T)[] opOnCall(T)(immutable(T[]) x) { return x; }

This is just a sketch, haven't thought it through, but if properly 
defined the scheme would work having built-in arrays work as they are 
today, and would also allow user-defined types to define their own 
opOnCall as they find fit.
Jun 22 2020
parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 22 June 2020 at 16:42:55 UTC, Andrei Alexandrescu 
wrote:

 On 6/22/20 12:14 PM, Paul Backus wrote:

 Making it work in general requires either subverting the type 
 system (like Rebindable does) or adding more special cases to 
 the language.


 The principled approach is to generalize the trick currently 
 used only for arrays. Consider the lowering:

 "Whenever something is passed to a function, .opOnCall is 
 appended."

 Then:

 auto ref T opOnCall(T)(auto ref T x) { return x; }
 const(T)[] opOnCall(T)(const(T[]) x) { return x; }
 immutable(T)[] opOnCall(T)(immutable(T[]) x) { return x; }

 This is just a sketch, haven't thought it through, but if 
 properly defined the scheme would work having built-in arrays 
 work as they are today, and would also allow user-defined types 
 to define their own opOnCall as they find fit.


The trick used for arrays does not only apply to function calls:

     const(int[]) a = [1, 2, 3];
     const(int)[] b = a; // compiles

IMHO the principled way to allow user-defined implicit 
conversions is...to allow user-defined implicit conversions. But 
iirc that's a can of worms Walter prefers not to open.
Jun 22 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 6/22/20 12:50 PM, Paul Backus wrote:

 On Monday, 22 June 2020 at 16:42:55 UTC, Andrei Alexandrescu wrote:

 On 6/22/20 12:14 PM, Paul Backus wrote:

 Making it work in general requires either subverting the type system 
 (like Rebindable does) or adding more special cases to the language.


 The principled approach is to generalize the trick currently used only 
 for arrays. Consider the lowering:

 "Whenever something is passed to a function, .opOnCall is appended."

 Then:

 auto ref T opOnCall(T)(auto ref T x) { return x; }
 const(T)[] opOnCall(T)(const(T[]) x) { return x; }
 immutable(T)[] opOnCall(T)(immutable(T[]) x) { return x; }

 This is just a sketch, haven't thought it through, but if properly 
 defined the scheme would work having built-in arrays work as they are 
 today, and would also allow user-defined types to define their own 
 opOnCall as they find fit.


 
 The trick used for arrays does not only apply to function calls:
 
      const(int[]) a = [1, 2, 3];
      const(int)[] b = a; // compiles


That's different - it's an implicit conversion.


 IMHO the principled way to allow user-defined implicit conversions 
 is...to allow user-defined implicit conversions. But iirc that's a can 
 of worms Walter prefers not to open.


What happens upon function calls is not an implicit conversion. It's a 
forced type change.
Jun 22 2020
next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 22 June 2020 at 21:46:51 UTC, Andrei Alexandrescu 
wrote:

 On 6/22/20 12:50 PM, Paul Backus wrote:

 
 The trick used for arrays does not only apply to function 
 calls:
 
      const(int[]) a = [1, 2, 3];
      const(int)[] b = a; // compiles


 That's different - it's an implicit conversion.


 IMHO the principled way to allow user-defined implicit 
 conversions is...to allow user-defined implicit conversions. 
 But iirc that's a can of worms Walter prefers not to open.


 What happens upon function calls is not an implicit conversion. 
 It's a forced type change.


So what you're saying is, it's even *less* principled than I 
thought? :)

Regardless, my broader point is that once we're open to the 
possibility of designing a new range API, all of this can be 
solved without any language changes by using an API that doesn't 
require mutation (i.e., tail()). Surely that's a better solution 
than implicitly inserting calls to arbitrary user-defined code 
every time someone passes an argument to a function.
Jun 22 2020
next sibling parent reply Aliak <something something.com> writes:
On Monday, 22 June 2020 at 22:17:43 UTC, Paul Backus wrote:

 On Monday, 22 June 2020 at 21:46:51 UTC, Andrei Alexandrescu 
 wrote:

 On 6/22/20 12:50 PM, Paul Backus wrote:

 
 The trick used for arrays does not only apply to function 
 calls:
 
      const(int[]) a = [1, 2, 3];
      const(int)[] b = a; // compiles


 That's different - it's an implicit conversion.


 IMHO the principled way to allow user-defined implicit 
 conversions is...to allow user-defined implicit conversions. 
 But iirc that's a can of worms Walter prefers not to open.


 What happens upon function calls is not an implicit 
 conversion. It's a forced type change.


 So what you're saying is, it's even *less* principled than I 
 thought? :)


Have you guys seen the work done here: 
https://gist.github.com/Biotronic/940f553fa1b4cbf80a98afe3d1970c6d ?

It is essentially the opOnCall with details on the library 
implementation as well.


 Regardless, my broader point is that once we're open to the 
 possibility of designing a new range API, all of this can be 
 solved without any language changes by using an API that 
 doesn't require mutation (i.e., tail()). Surely that's a better 
 solution than implicitly inserting calls to arbitrary 
 user-defined code every time someone passes an argument to a 
 function.


Can you use the tail() approach and keep the efficiency of 
popFront? - I.e. do you see that if tail is const it returns a 
newly constructed range and if tail() is mutable it can 
internally call a popFront links think and return a ref to itself?
Jun 22 2020
parent Paul Backus <snarwin gmail.com> writes:
On Monday, 22 June 2020 at 22:36:05 UTC, Aliak wrote:

 On Monday, 22 June 2020 at 22:17:43 UTC, Paul Backus wrote:

 Regardless, my broader point is that once we're open to the 
 possibility of designing a new range API, all of this can be 
 solved without any language changes by using an API that 
 doesn't require mutation (i.e., tail()). Surely that's a 
 better solution than implicitly inserting calls to arbitrary 
 user-defined code every time someone passes an argument to a 
 function.


 Can you use the tail() approach and keep the efficiency of 
 popFront? - I.e. do you see that if tail is const it returns a 
 newly constructed range and if tail() is mutable it can 
 internally call a popFront links think and return a ref to 
 itself?


In a world where tail() is a range primitive, popFront becomes an 
algorithm:

void popFront(R)(ref R r)
     if (isInputRangeV2!R && isMutable!R)
{
     r = r.tail;
}

Any decent optimizer should be able to turn this into a mutation. 
Since ranges already rely heavily on optimization to produce 
efficient code, I don't think it's unreasonable to rely on it 
here as well.

Of course, if you need more control, you are always free to 
specialize the algorithm for your own range type by defining a 
popFront method.
Jun 22 2020
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/22/20 6:17 PM, Paul Backus wrote:

 On Monday, 22 June 2020 at 21:46:51 UTC, Andrei Alexandrescu wrote:

 On 6/22/20 12:50 PM, Paul Backus wrote:

 The trick used for arrays does not only apply to function calls:

      const(int[]) a = [1, 2, 3];
      const(int)[] b = a; // compiles


 That's different - it's an implicit conversion.


 IMHO the principled way to allow user-defined implicit conversions 
 is...to allow user-defined implicit conversions. But iirc that's a 
 can of worms Walter prefers not to open.


 What happens upon function calls is not an implicit conversion. It's a 
 forced type change.


 
 So what you're saying is, it's even *less* principled than I thought? :)


Yeah, it's a hack, but it works very well.


 Regardless, my broader point is that once we're open to the possibility 
 of designing a new range API, all of this can be solved without any 
 language changes by using an API that doesn't require mutation (i.e., 
 tail()).


Using only tail() makes iteration with mutable ranges inefficient and 
iteration with immutable ranges difficult (must use recursion 
everywhere). Hardly a winner of a design contest.


  Surely that's a better solution than implicitly inserting calls 
 to arbitrary user-defined code every time someone passes an argument to 
 a function.


If there is, we haven't found it.
Jun 22 2020
parent reply Paul Backus <snarwin gmail.com> writes:
On Tuesday, 23 June 2020 at 05:15:49 UTC, Andrei Alexandrescu 
wrote:

 Regardless, my broader point is that once we're open to the 
 possibility of designing a new range API, all of this can be 
 solved without any language changes by using an API that 
 doesn't require mutation (i.e., tail()).


 Using only tail() makes iteration with mutable ranges 
 inefficient


I don't think it's a given that tail() is less efficient than 
popFront(). Here's a program where both tail() and popFront() 
produce the same assembly with `ldc -O2`:

https://d.godbolt.org/z/-S_JoF

Of course, if it does turn out to make a difference in some 
cases, mutable ranges are free to implement popFront as well. 
There will be a generic version in std.v2.algorithm to ensure 
that `mutableRange.popFront()` is always valid.


 and iteration with immutable ranges difficult (must use 
 recursion everywhere). Hardly a winner of a design contest.


Even if we assume for the sake of argument that "recursion == 
difficult", I would still say that "difficult" is an improvement 
over "impossible".
Jun 23 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/23/20 9:07 AM, Paul Backus wrote:

 On Tuesday, 23 June 2020 at 05:15:49 UTC, Andrei Alexandrescu wrote:

 Regardless, my broader point is that once we're open to the 
 possibility of designing a new range API, all of this can be solved 
 without any language changes by using an API that doesn't require 
 mutation (i.e., tail()).


 Using only tail() makes iteration with mutable ranges inefficient


 
 I don't think it's a given that tail() is less efficient than 
 popFront(). Here's a program where both tail() and popFront() produce 
 the same assembly with `ldc -O2`:
 
 https://d.godbolt.org/z/-S_JoF


With arrays, no problem. Two words to copy, easy to track, enregister 
etc. With elaborate ranges, definitely a problem.


 Of course, if it does turn out to make a difference in some cases, 
 mutable ranges are free to implement popFront as well. There will be a 
 generic version in std.v2.algorithm to ensure that 
 `mutableRange.popFront()` is always valid.
 

 and iteration with immutable ranges difficult (must use recursion 
 everywhere). Hardly a winner of a design contest.


 
 Even if we assume for the sake of argument that "recursion == 
 difficult", I would still say that "difficult" is an improvement over 
 "impossible".


There's "onerous" somewhere in there, too. I agree with Steve who said 
implementing recursive variants of most range algorithms in Phobos would 
be onerous.

The each() primitive I discussed may work nice actually because it 
allows recursion to be done in only itself, instead of every algorithm.
Jun 23 2020
next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Tuesday, 23 June 2020 at 16:20:26 UTC, Andrei Alexandrescu 
wrote:

 On 6/23/20 9:07 AM, Paul Backus wrote:

 I don't think it's a given that tail() is less efficient than 
 popFront(). Here's a program where both tail() and popFront() 
 produce the same assembly with `ldc -O2`:
 
 https://d.godbolt.org/z/-S_JoF


 With arrays, no problem. Two words to copy, easy to track, 
 enregister etc. With elaborate ranges, definitely a problem.


Do you (or anyone else reading this--feel free to chime in) have 
an example in mind of such an "elaborate range"? If so, I'd be 
happy to do the legwork of running additional experiments. No 
need to settle for speculation here when we can have actual data.


 Even if we assume for the sake of argument that "recursion == 
 difficult", I would still say that "difficult" is an 
 improvement over "impossible".


 There's "onerous" somewhere in there, too. I agree with Steve 
 who said implementing recursive variants of most range 
 algorithms in Phobos would be onerous.


Again, if you, Steve, or anyone else in this thread have any 
examples in mind, I'd be happy to run the experiment of writing 
recursive versions myself.


 The each() primitive I discussed may work nice actually because 
 it allows recursion to be done in only itself, instead of every 
 algorithm.


Even without making `each` a primitive, you can do this. 
Implement a few fundamental algorithms like `each`, `map`, 
`filter`, and `fold` using recursion, and many others can be 
built on top of them.
Jun 23 2020
next sibling parent Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Tuesday, 23 June 2020 at 21:19:39 UTC, Paul Backus wrote:

 Even without making `each` a primitive, you can do this. 
 Implement a few fundamental algorithms like `each`, `map`, 
 `filter`, and `fold` using recursion, and many others can be 
 built on top of them.


Slightly off-topic, but are you familiar with Clojure's 
transducers?
https://clojure.org/reference/transducers
https://www.youtube.com/watch?v=6mTbuzafcII

You can also check this talk from CppCon, to see how their 
applied to a statically typed language:
https://www.youtube.com/watch?v=6mTbuzafcII

For me, the main advantage is that this design allows 
implementing algorithms like map, filter, etc. without caring at 
all about the exact range interface. This for me is a big deal 
because, in my day-to-day job, I can't use much 
range/iterator/iterable/enumerable pull-style algorithms, because 
most my code is reactive/ asynchronous (push-based). So most of 
what I do is based on either Futures/Promises or Reactive 
Extensions:
http://reactivex.io/
http://reactivex.io/documentation/operators.html
Jun 24 2020
prev sibling parent reply Jon Degenhardt <jond noreply.com> writes:
On Tuesday, 23 June 2020 at 21:19:39 UTC, Paul Backus wrote:

 Do you (or anyone else reading this--feel free to chime in) 
 have an example in mind of such an "elaborate range"? If so, 
 I'd be happy to do the legwork of running additional 
 experiments. No need to settle for speculation here when we can 
 have actual data.


I don't think they'll satisfy what you are looking for, but I do 
have a few examples of ranges doing non-trivial things in public 
repos. You are welcome to take a look:

* bufferedByLine - 
https://github.com/eBay/tsv-utils/blob/master/common/src/tsv_utils/common/utils.d#L831
* inputSourceRange - 
https://github.com/eBay/tsv-utils/blob/master/common/src/tsv_utils/common/utils.d#L1443
* parseFieldList - 
https://github.com/eBay/tsv-utils/blob/master/common/src/tsv_utils/common/fieldlist.d#L291
* findFieldGroups - 
https://github.com/eBay/tsv-utils/blob/master/common/src/tsv_utils/common/fieldlist.d#L1045
* parseNumericFieldList - 
https://github.com/eBay/tsv-utils/blob/master/common/src/tsv_utils/common/fieldlist.d#L2029

None of them are used in performance sensitive regions of code, 
so they haven't been benchmarked or optimized.
Jun 24 2020
parent Jon Degenhardt <jond noreply.com> writes:
On Wednesday, 24 June 2020 at 17:00:37 UTC, Jon Degenhardt wrote:

 On Tuesday, 23 June 2020 at 21:19:39 UTC, Paul Backus wrote:

 Do you (or anyone else reading this--feel free to chime in) 
 have an example in mind of such an "elaborate range"? If so, 
 I'd be happy to do the legwork of running additional 
 experiments. No need to settle for speculation here when we 
 can have actual data.


 I don't think they'll satisfy what you are looking for, but I 
 do have a few examples of ranges doing non-trivial things in 
 public repos. You are welcome to take a look:

 * bufferedByLine - 
 https://github.com/eBay/tsv-utils/blob/master/common/src/tsv_utils/common/utils.d#L831
 * inputSourceRange - 
 https://github.com/eBay/tsv-utils/blob/master/common/src/tsv_utils/common/utils.d#L1443
 * parseFieldList - 
 https://github.com/eBay/tsv-utils/blob/master/common/src/tsv_utils/common/fieldlist.d#L291
 * findFieldGroups - 
 https://github.com/eBay/tsv-utils/blob/master/common/src/tsv_utils/common/fieldlist.d#L1045
 * parseNumericFieldList - 
 https://github.com/eBay/tsv-utils/blob/master/common/src/tsv_utils/common/fieldlist.d#L2029

 None of them are used in performance sensitive regions of code, 
 so they haven't been benchmarked or optimized.


A correction to the last entry. The line number in the URL is to 
a different range. Still a valid range to look at, but in the end 
it just returns a Phobos range. The one I meant to include is a 
more typical range. The correct entries, replacing the last one:

* parseNumericFieldGroup - 
https://github.com/eBay/tsv-utils/blob/master/common/src/tsv_utils/common/fieldlist.d#L2029
* parseNumericFieldList - 
https://github.com/eBay/tsv-utils/blob/master/common/src/tsv_utils/common/fieldlist.d#L2482
Jun 26 2020
prev sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Tuesday, 23 June 2020 at 16:20:26 UTC, Andrei Alexandrescu 
wrote:

 On 6/23/20 9:07 AM, Paul Backus wrote:

 I don't think it's a given that tail() is less efficient than 
 popFront(). Here's a program where both tail() and popFront() 
 produce the same assembly with `ldc -O2`:
 
 https://d.godbolt.org/z/-S_JoF


 With arrays, no problem. Two words to copy, easy to track, 
 enregister etc. With elaborate ranges, definitely a problem.

 ...

 There's "onerous" somewhere in there, too. I agree with Steve 
 who said implementing recursive variants of most range 
 algorithms in Phobos would be onerous.

 The each() primitive I discussed may work nice actually because 
 it allows recursion to be done in only itself, instead of every 
 algorithm.


I'm struck that no one AFAICS has suggested the following 
alternative: instead of `tail`, to allow popFront() (and if 
implemented, popBack()) to return an `Option!(ElementType, None)`.

What is returned would be either the popped element, or None if 
no more elements remain (with the nice by-product of no longer 
getting assert failures if pop{Front,Back} are called when the 
range is empty).

This might allow some natural API evolution along the following 
lines:

   * an input range need only implement such an option-based 
popFront() and
     no other methods

     - for backwards compatibility, we could also support 
implementing `front`
       and a `void`-returning popFront()

   * a forward range would additionally implement `front` and 
`save` methods:

     - `front` because, since a forward range is deterministic, it 
should
       always be possible to know up front (pun intended...) what 
the first
       element should be

     - `save` for the usual reasons

   * all other range types should derive naturally from this

     - we might want to consider a more fine-grained approach to 
bidirectional
       ranges, e.g. allowing a bidirectional input range with 
option-returning
       popFront and popBack

   * over time we could/should deprecate void-returning 
pop{Front,Back}, and
     possibly also defining `front` or `back` if `save` is not 
also defined

The approach of having only option-returning popFront would be 
particularly useful for input ranges where the front cannot or 
should not be predefined on construction (e.g. RNGs, or stuff 
that wraps them like RandomSample and friends).

(Is there actually a good example of an input range where the 
first element can be predefined on construction in a way that 
isn't -- like pseudo-RNGs -- an implementation detail that one 
probably wants to hide from the user?)
Jun 24 2020
next sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 24 June 2020 at 14:17:17 UTC, Joseph Rushton 
Wakeling wrote:

 [snip]

 I'm struck that no one AFAICS has suggested the following 
 alternative: instead of `tail`, to allow popFront() (and if 
 implemented, popBack()) to return an `Option!(ElementType, 
 None)`.

 [snip]


This is similar to what I had said previously [1] about cursors 
(though Steven disagreed with my initial take). However, it 
sounds like an Option is more elegant.

[1] 
https://forum.dlang.org/post/nudkmmtoeqtznuorneev forum.dlang.org
Jun 24 2020
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/24/20 10:17 AM, Joseph Rushton Wakeling wrote:

 
 I'm struck that no one AFAICS has suggested the following alternative: 
 instead of `tail`, to allow popFront() (and if implemented, popBack()) 
 to return an `Option!(ElementType, None)`.
 
 What is returned would be either the popped element, or None if no more 
 elements remain (with the nice by-product of no longer getting assert 
 failures if pop{Front,Back} are called when the range is empty).


I think the discussion about tail() had to do with immutable ranges. 
Having popXxx() return something would still have it mutate the range.
Jun 24 2020
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Wednesday, 24 June 2020 at 16:08:05 UTC, Andrei Alexandrescu 
wrote:

 On 6/24/20 10:17 AM, Joseph Rushton Wakeling wrote:

 I'm struck that no one AFAICS has suggested the following 
 alternative: instead of `tail`, to allow popFront() (and if 
 implemented, popBack()) to return an `Option!(ElementType, 
 None)`.
 
 What is returned would be either the popped element, or None 
 if no more elements remain (with the nice by-product of no 
 longer getting assert failures if pop{Front,Back} are called 
 when the range is empty).


 I think the discussion about tail() had to do with immutable 
 ranges. Having popXxx() return something would still have it 
 mutate the range.


Ah, gotcha, thanks, I'd missed that context.

Nevertheless, I'd like the option-returning popFront to be 
considered on its own merits, because it seems to me to add 
useful distinctions between what input and forward ranges to do.

On immutability: it feels like this relates at least in part to 
not necessarily distinguishing adequately between ranges versus 
containers (or more generally, ranges versus owners of memory).

Have I missed anything in noting that ranges can be divided 
roughly along the following lines:

   - processing external input, where the internal state is just 
some small
     memory buffer or other collection of variables used to store 
the latest
     data read

   - generative mechanisms where the range elements are produced 
by a sequential
     algorithm

   - iteration across data stored in memory

... where immutability only really makes sense in the first place 
w.r.t. the last of these?  And there, it doesn't really make 
sense that the range be const or immutable -- only that the 
underlying data is?
Jun 26 2020
parent reply Paul Backus <snarwin gmail.com> writes:
On Friday, 26 June 2020 at 14:46:24 UTC, Joseph Rushton Wakeling 
wrote:

 Have I missed anything in noting that ranges can be divided 
 roughly along the following lines:

   - processing external input, where the internal state is just 
 some small
     memory buffer or other collection of variables used to 
 store the latest
     data read

   - generative mechanisms where the range elements are produced 
 by a sequential
     algorithm

   - iteration across data stored in memory

 ... where immutability only really makes sense in the first 
 place w.r.t. the last of these?  And there, it doesn't really 
 make sense that the range be const or immutable -- only that 
 the underlying data is?


Because const and immutable are transitive, any data structure 
that includes a range, either directly or by reference, is itself 
forced to be mutable. This makes D's const and immutable 
significantly less useful than they otherwise would be; i.e., it 
makes D a worse language.

The fact that you, personally, cannot think of a use-case for 
immutable ranges off the top of your head is not a strong 
argument against supporting them. All language and library 
features ought to work together unless there is a specific reason 
for them not to (as is the case, for example, with garbage 
collection and BetterC).

So I think the burden of proof is on you here. If we're designing 
a new range interface, is there a good reason it *shouldn't* work 
with const and immutable?

One argument in favor of requiring mutability is that using 
tail() instead of popFront() could result in a loss of 
performance. I am working on investigating that claim using the 
examples provided by Jon Degenhardt earlier in this thread. 
Another, which I find less convincing, is that writing algorithms 
recursively instead of iteratively would be too "difficult" or 
"onerous". If you have any more to add, I am interested in 
hearing them.

[1] https://issues.dlang.org/show_bug.cgi?id=5377
Jun 26 2020
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Friday, 26 June 2020 at 16:03:23 UTC, Paul Backus wrote:

 The fact that you, personally, cannot think of a use-case for 
 immutable ranges off the top of your head is not a strong 
 argument against supporting them. All language and library 
 features ought to work together unless there is a specific 
 reason for them not to (as is the case, for example, with 
 garbage collection and BetterC).

 So I think the burden of proof is on you here. If we're 
 designing a new range interface, is there a good reason it 
 *shouldn't* work with const and immutable?


There's no need to be combative here.  I'm not trying to argue a 
side, but to make sure we get clarity on what it means for ranges 
to be compatible with const and immutable in this way.

Whether we like it or not, pretty much all ranges currently work 
via some under-the-hood mutability, whether it's just updating 
the pointer of a slice, incrementing an internal variable, or 
reading from some external input into an internal buffer.

In some cases there are ready ways out of that.  For a forward 
range whose internal state is cheap to duplicate, there should be 
no problem just (automatically) making a mutable deep copy to 
iterate over.  If we're talking about a range that iterates over 
the contents of some memory locations, it seems natural enough 
that it should be possible to automatically create a mutable 
shallow copy that iterates over immutable memory locations 
(analogous to immutable(T[]) => immutable(T)[] and so on).

But -- for example -- what would it mean to have an input range 
as part of a const or immutable data structure?  Forget the 
strict interpretation of the current D keywords, let's just think 
conceptually for a moment.  Do we want to think of an immutable 
input range as just a source of input, with any under-the-hood 
mutability just an implementation detail that shouldn't be of 
interest to the external caller?  Do we consider it a 
contradiction in terms?

I'm inclined to agree with the suggestion that input ranges 
should either be non-copyable or have reference semantics, but I 
don't think that really resolves the issue given the 
transitiveness of const and immutable.

I also think it may be worth making a point of comparison to 
Rust's iterators.  Obviously Rust has a different approach to 
immutability than D -- the main concern is preventing multiple 
mutable references to the same piece of data -- but there is a 
very clear separation between iterators versus what they iterate 
over, and generally speaking (probably because of that 
separation) iterators themselves seem to be always implemented as 
mutable.


 One argument in favor of requiring mutability is that using 
 tail() instead of popFront() could result in a loss of 
 performance. I am working on investigating that claim using the 
 examples provided by Jon Degenhardt earlier in this thread.


FWIW my objection to `tail` was probably misplaced.  What I was 
interested in was coming from a completely different angle to the 
const/immutable concern.

Specifically, I'm interested in drawing a clear line between 
input ranges -- where it is not really a given that `front` can 
be pre-populated -- versus forward ranges.  For example, given a 
RandomSample, one arguably does not want `front` to be 
pre-determined on construction, but from the moment one starts 
consuming the range.  Given the current range API that means some 
unpleasant is-this-the-first-call special-casing under the hood 
of the `front` method.

So, what I'm interested in is whether we can reliably rework the 
input range API such that the only way to determine the next 
element is to actually fetch it.  That would also work with a 
`tail`-like approach if the function concerned would return 
something like:

     Tuple!(Option!(ElementType, None), Tail)

... but I'm not sure if that would interfere with other concerns 
about that approach (e.g. would it get in the way of optimizing 
performance of the recursive approach?).


 Another, which I find less convincing, is that writing 
 algorithms recursively instead of iteratively would be too 
 "difficult" or "onerous". If you have any more to add, I am 
 interested in hearing them.


I don't have specific examples, but it's worth noting that one of 
the plus points of D is that it tends to allow you to write any 
given code in the simplest and most natural way, rather than 
forcing you through more complicated paradigms for the sake of it.

So, I'd personally be reluctant to force all writers of ranges to 
write their code in a more complicated/virtuous way unless their 
use case specifically requires it.

With that in mind, it might be better to identify ways to 
simplify the _implementation_ of ranges that need to support 
const/immutable use cases.  That is, keep it pay-as-you-go -- you 
don't have to write ranges in a way that supports const/immutable 
if you don't have that use-case -- but if you do, the cost you 
have to pay is less.
Jun 27 2020
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 27 June 2020 at 12:25:34 UTC, Joseph Rushton 
Wakeling wrote:


 But -- for example -- what would it mean to have an input range 
 as part of a const or immutable data structure?  Forget the 
 strict interpretation of the current D keywords, let's just 
 think conceptually for a moment.  Do we want to think of an 
 immutable input range as just a source of input, with any 
 under-the-hood mutability just an implementation detail that 
 shouldn't be of interest to the external caller?  Do we 
 consider it a contradiction in terms?


An immutable or const input range just cannot be. An input range 
can't not be mutable.
If a range itself can implement a `tail()`, it's not an input 
range, it's a forward range. The only way for `tail()` to work 
with input ranges is to be a free function:

R tail(R)(R r)
if (isInputRangeWithAHypotheticalAPI!R)
{
     r.next(); // fetchNext(), advance(), whatever the API is
     return move(r);
}

and used like this:

input = move(input).tail;

(Yes, I'm still insisting that input ranges must be non-copyable).


 I also think it may be worth making a point of comparison to 
 Rust's iterators.  Obviously Rust has a different approach to 
 immutability than D -- the main concern is preventing multiple 
 mutable references to the same piece of data -- but there is a 
 very clear separation between iterators versus what they 
 iterate over, and generally speaking (probably because of that 
 separation) iterators themselves seem to be always implemented 
 as mutable.


True input ranges are basically Rust's iterators, i.e. their API 
should be

Optional!T next(); // in other words, a bool and a T

or variations thereof, as outlined by Andrei some pages back. 
IMHO, the above API is the cleanest. But there is also place for 
buffered input ranges (i.e. the currently existing API), see 
below.


 So, what I'm interested in is whether we can reliably rework 
 the input range API such that the only way to determine the 
 next element is to actually fetch it.


Both APIs have their place. A "true" input range (i.e. a 
generator) yields temporaries, and only needs a "fetch next" 
primitive. But the *algorithms* may require a buffer. Consider a 
`find`: it needs to return the needle and the remainder of the 
range. In order to find the needle it needs to pop it off from 
the "fetch next" haystack, irreversibly advancing it. If we want 
to keep `find` generic over input and forward ranges (i.e. 
returning a range), it will have to wrap the haystack into a 
buffered input range with the front/popFront API, placing the 
found needle at the front, i.e. hypothetically:

auto find(alias pred, Range, Needle)(Range range, auto ref const 
Needle needle)
if (isInputRangeWithAHypotheticalAPI!R)
{
     static struct Result
     {
         private typeof(Range.init.next()) buf;
         private Range src;

         // So long as the Optional holds a valid value, range is 
not empty
          property bool empty() const { return !buf; }

         // Accessing an empty Optional would be illegal,
         // so the non-empty() invariant is observed.
         ref front() inout { return buf.value; }

         void popFront() { buf = src.next(); }
     }

     typeof(range.next()) current; // i.e. an Optional!ElementType
     while (true)
     {
         current = range.next;
         if (!current || pred(current.value, needle))
             break;
     }

     return Result(move(current), move(range));
}
Jun 27 2020
parent reply Paul Backus <snarwin gmail.com> writes:
On Saturday, 27 June 2020 at 13:11:09 UTC, Stanislav Blinov wrote:

 On Saturday, 27 June 2020 at 12:25:34 UTC, Joseph Rushton 
 Wakeling wrote:


 But -- for example -- what would it mean to have an input 
 range as part of a const or immutable data structure?  Forget 
 the strict interpretation of the current D keywords, let's 
 just think conceptually for a moment.  Do we want to think of 
 an immutable input range as just a source of input, with any 
 under-the-hood mutability just an implementation detail that 
 shouldn't be of interest to the external caller?  Do we 
 consider it a contradiction in terms?


 An immutable or const input range just cannot be. An input 
 range can't not be mutable.
 If a range itself can implement a `tail()`, it's not an input 
 range, it's a forward range.


This is false.

While it is true that a pure input range cannot exist without the 
presence of mutable state *somewhere*, it does not follow that 
the mutable state must be stored in (or pointed to by) the range 
itself. Consider the following example:

     auto fdByChar(int fd)
     {
         import core.sys.posix.unistd;

         static struct Result
         {
             private int fd;
             bool empty;
             char front;

             Result tail() const
                 in (!empty)
             {
                 char next;
                 ssize_t nread = read(fd, cast(void*) &next, 1);

                 if (nread == 0)
                     return Result(fd, true, char.init);
                 else if (nread == 1)
                     return Result(fd, false, next);
                 else
                     throw new Exception("Error reading file");
             }
         }

         return Result(fd, false, char.init).tail;
     }


This is a pure input range. Copying it and iterating the copy 
will invalidate the original. Yet it is perfectly capable of 
being `immutable`. Here's an example of usage:

     void each(alias fun, Range)(Range range)
     {
         if (range.empty) return;
         fun(range.front);
         range.tail.each!fun;
     }

     void main()
     {
         import std.stdio;
         immutable stdinByChar = fdByChar(0);
         stdinByChar.each!writeln;
     }


Of course, there is mutable state involved here, but that state 
is not in the range--it's in the kernel. So the `immutable` 
qualifier on the range does not apply to it.
Jun 27 2020
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 27 June 2020 at 17:16:10 UTC, Paul Backus wrote:


 An immutable or const input range just cannot be. An input 
 range can't not be mutable.
 If a range itself can implement a `tail()`, it's not an input 
 range, it's a forward range.


 This is false.

 While it is true that a pure input range cannot exist without 
 the presence of mutable state *somewhere*, it does not follow 
 that the mutable state must be stored in (or pointed to by) the 
 range itself. Consider the following example:

     auto fdByChar(int fd)
 ...
 Of course, there is mutable state involved here, but that state 
 is not in the range--it's in the kernel. So the `immutable` 
 qualifier on the range does not apply to it.


Thereby result of (input.tail == input.tail) is a coin flip? I 
don't think that's a sensible design.
Jun 27 2020
parent reply Paul Backus <snarwin gmail.com> writes:
On Saturday, 27 June 2020 at 17:30:54 UTC, Stanislav Blinov wrote:

 Thereby result of (input.tail == input.tail) is a coin flip? I 
 don't think that's a sensible design.


Equality is not part of the range interface (current or 
proposed), so there is no guarantee that comparing two ranges 
with `==` will give you a useful result.
Jun 27 2020
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 27 June 2020 at 18:55:39 UTC, Paul Backus wrote:

 On Saturday, 27 June 2020 at 17:30:54 UTC, Stanislav Blinov 
 wrote:

 Thereby result of (input.tail == input.tail) is a coin flip? I 
 don't think that's a sensible design.


 Equality is not part of the range interface (current or 
 proposed), so there is no guarantee that comparing two ranges 
 with `==` will give you a useful result.


Equality is very much part of the struct interface though. I'll 
restate. What about input.tail.empty == input.tail.empty? Or 
input.tail.front == input.tail.front? Or, speaking of ranges, a 
input.tail.equal(input.tail)?..
I thought you were concerned about having to explain something to 
beginners ;)

If the explanation is "`tail() const` is impure, and calling it 
repeatedly with the same `this` produces different results, even 
when `this` is immutable", "WTF???" would be quite a reasonable 
follow-up question. This is not malloc. If we do define a `tail` 
primitive, it better yield the same output for the same input, 
meaning it would only be a primitive of forward ranges.
Jun 27 2020
next sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Saturday, 27 June 2020 at 20:14:36 UTC, Stanislav Blinov wrote:

 Equality is very much part of the struct interface though. I'll 
 restate. What about input.tail.empty == input.tail.empty? Or 
 input.tail.front == input.tail.front? Or, speaking of ranges, a 
 input.tail.equal(input.tail)?..


Does it even make _sense_ to be able to call `tail` twice on the 
same input range instance?


 If the explanation is "`tail() const` is impure, and calling it 
 repeatedly with the same `this` produces different results,
 even when `this` is immutable", "WTF???" would be quite a 
 reasonable follow-up question.


One of the problems with thinking about ranges -- and the current 
range API reflects this -- is that it tends to start by thinking 
about deterministic examples (e.g. arrays).  But while some input 
ranges may be deterministic in practice, it is not possible to 
reasonably assume "same input, same output" behaviour.  That's 
the whole _point_ of input ranges.

As an example: reading a stream of values from a crypto RNG, or a 
"true random" source measured from some ongoing physical process.

A key thing to observe there is that, immutable or no, the state 
of `this` doesn't give you complete information about what comes 
next in the sequence, and indeed the result may also depend on 
precisely _when_ the next element is requested (e.g. if my input 
range is a "true random" noise source, I'm likely to get a 
different value if I wait 0.1 sec to call it rather than calling 
it right now, even if no one else has been using that noise 
source).

Conceptually that doesn't fit very well with the concept of 
`tail`, but that's just a matter of a better name.  The real 
problem here isn't the indeterministic results, it's the idea 
that the original `this` is usable after `tail` has been called 
on it that is the problem.
Jun 28 2020
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Sunday, 28 June 2020 at 10:42:14 UTC, Joseph Rushton Wakeling 
wrote:

 On Saturday, 27 June 2020 at 20:14:36 UTC, Stanislav Blinov 
 wrote:

 Equality is very much part of the struct interface though. 
 I'll restate. What about input.tail.empty == input.tail.empty? 
 Or input.tail.front == input.tail.front? Or, speaking of 
 ranges, a input.tail.equal(input.tail)?..


 Does it even make _sense_ to be able to call `tail` twice on 
 the same input range instance?


No, it does not, that's my point. "Don't do that" isn't the 
answer. "Can't do that" (statically disallowed, not present in 
the interface) is the answer.


 If the explanation is "`tail() const` is impure, and calling 
 it repeatedly with the same `this` produces different results,
 even when `this` is immutable", "WTF???" would be quite a 
 reasonable follow-up question.


 One of the problems with thinking about ranges -- and the 
 current range API reflects this -- is that it tends to start by 
 thinking about deterministic examples (e.g. arrays).


Arrays are random access ranges (which are also forward). They're 
not input ranges :)
The current range API incorrectly treats any range as at least an 
input range. If we are talking about making changes to range API 
- that is one of the changes that needs to be made.


 But while some input ranges may be deterministic in practice, 
 it is not possible to reasonably assume "same input, same 
 output" behaviour.  That's the whole _point_ of input ranges.


Yes, that is why an input range cannot be implementing `tail`. 
You cannot keep the original range *and* the tail. That's what 
forward ranges do.


 As an example: reading a stream of values from a crypto RNG, or 
 a "true random" source measured from some ongoing physical 
 process.

 A key thing to observe there is that, immutable or no, the 
 state of `this` doesn't give you complete information about 
 what comes next in the sequence, and indeed the result may also 
 depend on precisely _when_ the next element is requested (e.g. 
 if my input range is a "true random" noise source, I'm likely 
 to get a different value if I wait 0.1 sec to call it rather 
 than calling it right now, even if no one else has been using 
 that noise source).



 Conceptually that doesn't fit very well with the concept of 
 `tail`, but that's just a matter of a better name.


It isn't. It's a matter of semantics of the range. You can only 
consume or discard an input range. You can't fork it. Conversely, 
if you *can* fork a range - it's a forward range. Therefore, an 
input range cannot have a `tail` :)


 The real problem here isn't the indeterministic results, it's 
 the idea that the original `this` is usable after `tail` has 
 been called on it that is the problem.


We cannot, in the language, statically, express a "usability" of 
an lvalue.
Jun 28 2020
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Sunday, 28 June 2020 at 11:33:39 UTC, Stanislav Blinov wrote:

 Does it even make _sense_ to be able to call `tail` twice on 
 the same input range instance?


 No, it does not, that's my point. "Don't do that" isn't the 
 answer. "Can't do that" (statically disallowed, not present in 
 the interface) is the answer.


Indeed.  You seem to be thinking I'm disagreeing with you, when 
actually we are in agreement on this point ;-)


 One of the problems with thinking about ranges -- and the 
 current range API reflects this -- is that it tends to start 
 by thinking about deterministic examples (e.g. arrays).


 Arrays are random access ranges (which are also forward). 
 They're not input ranges :)


That's not strictly true: one can reasonably _interpret_ an array 
as an input range if that's what is desired.  What makes an input 
range distinct is not whether the actual elements come from some 
impure input source (it can be deterministic under the hood, 
whether from iterating over array elements, or a deterministic 
generative mechanism like a PRNG, or whatever), but what 
assumptions the caller can make about how to use and interact 
with it.


 The current range API incorrectly treats any range as at least 
 an input range. If we are talking about making changes to range 
 API - that is one of the changes that needs to be made.


That doesn't really make sense to me.  Why is it not possible to 
interact with any other range via input range semantics?


 But while some input ranges may be deterministic in practice, 
 it is not possible to reasonably assume "same input, same 
 output" behaviour.  That's the whole _point_ of input ranges.


 Yes, that is why an input range cannot be implementing `tail`. 
 You cannot keep the original range *and* the tail. That's what 
 forward ranges do.


Right, _you cannot keep the original range and the tail_.  But 
that doesn't block you having a method which takes the original 
range as input, and returns the updated range.  The issue is 
stopping you using the _original_ range (e.g. requiring it to be 
passed into the method via a move rather than by reference?).
Jun 28 2020
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Sunday, 28 June 2020 at 12:43:53 UTC, Joseph Rushton Wakeling 
wrote:

 On Sunday, 28 June 2020 at 11:33:39 UTC, Stanislav Blinov wrote:

 Does it even make _sense_ to be able to call `tail` twice on 
 the same input range instance?


 No, it does not, that's my point. "Don't do that" isn't the 
 answer. "Can't do that" (statically disallowed, not present in 
 the interface) is the answer.


 Indeed.  You seem to be thinking I'm disagreeing with you, when 
 actually we are in agreement on this point ;-)


Ah, that wasn't evident from the question :*)


 One of the problems with thinking about ranges -- and the 
 current range API reflects this -- is that it tends to start 
 by thinking about deterministic examples (e.g. arrays).


 Arrays are random access ranges (which are also forward). 
 They're not input ranges :)


 That's not strictly true: one can reasonably _interpret_ an 
 array as an input range if that's what is desired.


That does not make arrays input ranges. They remain random-access 
ranges.


 What makes an input range distinct is not whether the actual 
 elements come from some impure input source (it can be 
 deterministic under the hood, whether from iterating over array 
 elements, or a deterministic generative mechanism like a PRNG, 
 or whatever), but what assumptions the caller can make about 
 how to use and interact with it.


That would be design by convention, which I'm strongly opposed 
to, and arguing against. There's no need to *document* caller's 
assumptions when we can devise an API that drives them. What 
makes an input range distinct is that, unlike other kinds, it can 
only be consumed once - there's no need for any other 
assumptions. The current range API does not enforce this in any 
way, hence the copying conundrum and Jonathan's "don't use 
original after you make a copy" or other similar statements that 
amount to "you should know what you're doing".


 The current range API incorrectly treats any range as at least 
 an input range. If we are talking about making changes to 
 range API - that is one of the changes that needs to be made.


 That doesn't really make sense to me.  Why is it not possible 
 to interact with any other range via input range semantics?


Because they have different semantics! The current range API 
attempts to differentiate solely on interface (which can indeed 
appear intersecting) but not semantics. What I meant is, today in 
Phobos, there are algorithms that do this:

auto algo(Range)(Range range)
if (isInputRange!Range)
{
     /* ... */
     static if (isForwardRange!Range)
     {
         /* ... */
     }
}

Input ranges are treated as a subset of forward ranges. But 
they're not - they're distinct, non-intersecting sets. If we do 
change the API, the above will have to become either (depending 
on the algorithm):

auto algo(Range)(Range range)
if (isInputRange!Range)
{ /* ... */ }

auto algo(Range)(Range range)
if (isForwardRange!Range)
{ /* ... */ }

// OR, given a hypothetical `isSomeRange` test

auto algo(Range)(Range range)
if (isSomeRange!Range) // note: isSomeRange, not isInputRange
{
     /* use common API, if any ... */
     static if (isForwardRange!Range)
     {
         /* ... */
     }
}


 Right, _you cannot keep the original range and the tail_.  But 
 that doesn't block you having a method which takes the original 
 range as input, and returns the updated range.  The issue is 
 stopping you using the _original_ range (e.g. requiring it to 
 be passed into the method via a move rather than by reference?).


Yes, i.e. the free function implementation of `tail` that I 
posted previously (which would work correctly for both input 
ranges and forward ranges, *assuming* input ranges aren't 
copyable and forward ranges are).
Jun 28 2020
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sun, Jun 28, 2020 at 02:00:42PM +0000, Stanislav Blinov via Digitalmars-d
wrote:
[...]

 What makes an input range distinct is that, unlike other kinds, it can
 only be consumed once - there's no need for any other assumptions. The
 current range API does not enforce this in any way, hence the copying
 conundrum and Jonathan's "don't use original after you make a copy" or
 other similar statements that amount to "you should know what you're
 doing".


This is not a sufficient assumption, because there remains the question
of what state a range should be in after you iterate over it partially,
say, using a foreach loop:

	MyRange r = ...;
	foreach (e; r) {
		if (someCondition) break;
	}
	foreach (e; r) {
		// Do we get the remaining elements of r here, or do we
		// get all the elements from the beginning again?
	}

Even if we grant that r is a forward range, the semantics of the above
code is still underspecified.

And it's not just a matter of specifying that the semantics should be;
there's also the question of whether we should specify anything at all,
keeping in mind that the more specific the definition, the less room
there is for efficient implementation of ranges (i.e., some
optimizations may be excluded if there are too many semantic
requirements on ranges).


[...]

 Input ranges are treated as a subset of forward ranges. But they're
 not - they're distinct, non-intersecting sets.


OTOH, there is a great deal of advantage in using the subset of
semantics in which input ranges *are* subsets of forward ranges, because
this allows us to unify a lot of iteration logic.  Otherwise, like you
say below, every algorithm will need to be duplicated, once for input
ranges, once for forward ranges, which will instantly double the size of
Phobos algorithms.


[...]

 auto algo(Range)(Range range)
 if (isInputRange!Range)
 { /* ... */ }
 
 auto algo(Range)(Range range)
 if (isForwardRange!Range)
 { /* ... */ }


[...]


T

-- 
If it tastes good, it's probably bad for you.
Jun 28 2020
parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Sunday, 28 June 2020 at 15:07:21 UTC, H. S. Teoh wrote:

 On Sun, Jun 28, 2020 at 02:00:42PM +0000, Stanislav Blinov via 
 Digitalmars-d wrote: [...]

 What makes an input range distinct is that, unlike other 
 kinds, it can only be consumed once - there's no need for any 
 other assumptions. The current range API does not enforce this 
 in any way, hence the copying conundrum and Jonathan's "don't 
 use original after you make a copy" or other similar 
 statements that amount to "you should know what you're doing".


 This is not a sufficient assumption, because there remains the 
 question of what state a range should be in after you iterate 
 over it partially, say, using a foreach loop:

 	MyRange r = ...;
 	foreach (e; r) {
 		if (someCondition) break;
 	}
 	foreach (e; r) {
 		// Do we get the remaining elements of r here, or do we
 		// get all the elements from the beginning again?
 	}



 Even if we grant that r is a forward range, the semantics of 
 the above code is still underspecified.


Of course it isn't, because there's a problem with `foreach`. It 
iterates over a copy, except if there's an opApply. I'm not sure 
what your argument is. I repeat: "The current range API does not 
enforce [input range behavior] in any way, hence the copying 
conundrum..." `foreach` is very much a part of range API.


 And it's not just a matter of specifying that the semantics 
 should be; there's also the question of whether we should 
 specify anything at all, keeping in mind that the more specific 
 the definition, the less room there is for efficient 
 implementation of ranges (i.e., some optimizations may be 
 excluded if there are too many semantic requirements on ranges).


Yes, we should. For example, non-copyability of input ranges, 
which enables *more* efficient implementations (whereas expecting 
them to remain copyable dictates the opposite).


 ...there is a great deal of advantage in using the subset of 
 semantics in which input ranges *are* subsets of forward 
 ranges, because this allows us to unify a lot of iteration 
 logic.


That's not based on range semantics, it's based solely on 
interface. An input range and a forward range can have the same 
interface for iteration, i.e. a `fetchNext` or its variant. The 
only difference is that you can't fork the former.


 Otherwise, like you say below, every algorithm will need to be 
 duplicated, once for input ranges, once for forward ranges, 
 which will instantly double the size of Phobos algorithms.


That's a gross exaggerration. Algorithms already do the semantic 
duplication, for arrays, for hasLvalueElements, etc., etc., 
ultimately, when everything else fails, arriving at the "while 
(!r.empty) r.popFront()".
Jun 28 2020
prev sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Saturday, 27 June 2020 at 20:14:36 UTC, Stanislav Blinov wrote:

 If the explanation is "`tail() const` is impure, and calling it 
 repeatedly with the same `this` produces different results, 
 even when `this` is immutable", "WTF???" would be quite a 
 reasonable follow-up question. This is not malloc. If we do 
 define a `tail` primitive, it better yield the same output for 
 the same input, meaning it would only be a primitive of forward 
 ranges.


It sounds like what you are really trying to say here is that 
input ranges and forward ranges should not use the same 
interface, and that input ranges should instead implement only 
`Option!T next()`. I agree that this would be ideal, and that if 
this change were made, tail() would be required only for forward 
ranges and up. In that case, requiring tail() to be pure would 
make sense.

My example was a response to the specific claim that "an 
immutable input range cannot exist." Specifically, it was a 
counterexample demonstrating that the claim is false. As I'm sure 
you can see, the claim is still false even if we agree to use 
next() instead of tail() for input ranges, since next() is 
obviously allowed to be impure.
Jun 28 2020
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Sunday, 28 June 2020 at 14:12:39 UTC, Paul Backus wrote:


 It sounds like what you are really trying to say here is that 
 input ranges and forward ranges should not use the same 
 interface, and that input ranges should instead implement only 
 `Option!T next()`.


Not exactly. A given forward range may also implement such a 
`next`. Taking your example, with a seekable file and `pread`. 
Which would, of course, narrow the set of "files" it would be 
able to work with. A forward range need not, necessarily, have a 
`front` and a `popFront` (see that funky List that Andrei posted 
a while back). We should differentiate by some other artifact. I 
maintain that artifact should be copyability. Or we can keep the 
distinction based on the presence of save(), and live forever 
with the ambiguous semantics and the "don't use original after 
copy unless you know what you're doing" convention (yuck). Are 
there even other options besides accepting UB?


 My example was a response to the specific claim that "an 
 immutable input range cannot exist." Specifically, it was a 
 counterexample demonstrating that the claim is false. As I'm 
 sure you can see, the claim is still false even if we agree to 
 use next() instead of tail() for input ranges, since next() is 
 obviously allowed to be impure.


I can't see that. I don't see how you're falsifying that claim 
when in order to implement the range you need mutable state. 
"It's mutable, but the compiler can't see that, therefore it's 
immutable" - that is a dangerous justification. There are indeed 
some legitimate applications for it (allocators, mutexes), but 
this isn't one of them. If `a.tail` is not returning the tail of 
`a`, then it's either a bug in the design (shouldn't be called 
`tail`) or in the implementation (should not mutate *any* state).
Jun 28 2020
parent reply Paul Backus <snarwin gmail.com> writes:
On Sunday, 28 June 2020 at 15:33:48 UTC, Stanislav Blinov wrote:

 On Sunday, 28 June 2020 at 14:12:39 UTC, Paul Backus wrote:


 It sounds like what you are really trying to say here is that 
 input ranges and forward ranges should not use the same 
 interface, and that input ranges should instead implement only 
 `Option!T next()`.


 Not exactly. A given forward range may also implement such a 
 `next`. [...] We should differentiate by some other artifact. I 
 maintain that artifact should be copyability. Or we can keep 
 the distinction based on the presence of save(), and live 
 forever with the ambiguous semantics and the "don't use 
 original after copy unless you know what you're doing" 
 convention (yuck). Are there even other options besides 
 accepting UB?


If input ranges are only required to implement next(), then the 
distinction will be that forward ranges implement head()/tail() 
and input ranges don't. (In this scenario, head() and tail() will 
of course be required to return the same result if called 
multiple times.)

For *some* forward ranges, it will be possible to implement 
next() as well, in which case those ranges will be capable of 
functioning as both input ranges and forward ranges. I don't see 
anything wrong with that.


 My example was a response to the specific claim that "an 
 immutable input range cannot exist." Specifically, it was a 
 counterexample demonstrating that the claim is false. As I'm 
 sure you can see, the claim is still false even if we agree to 
 use next() instead of tail() for input ranges, since next() is 
 obviously allowed to be impure.


 I can't see that. I don't see how you're falsifying that claim 
 when in order to implement the range you need mutable state.


Ok, I'll show you. Here's the same range implemented using next() 
instead of front() and tail():

     // Just enough for the example to compile
     struct Optional(T)
     {
         bool empty;
         T value;
     }

     Optional!T some(T)(T value)
     {
         return Optional!T(false, value);
     }

     Optional!T no(T)()
     {
         return Optional!T(true, T.init);
     }

     auto fdByChar(int fd)
     {
         import core.sys.posix.unistd;

         static struct Result
         {
             private int fd;

             Optional!char next() const
             {
                 char c;
                 ssize_t nread = read(fd, cast(void*) &c, 1);

                 if (nread == 0)
                     return no!char;
                 else if (nread == 1)
                     return some(c);
                 else
                     throw new Exception("Error reading file");
             }
         }

         return Result(fd);
     }

And here's the usage:

     void each(alias fun, Range)(Range range)
     {
         for (auto e = range.next; !e.empty; e = range.next)
             fun(e.value);
     }

     void main()
     {
         import std.stdio;
         immutable stdinByChar = fdByChar(0);
         stdinByChar.each!writeln;
     }

Once again, `stdinByChar` is an input range, and it is immutable. 
QED.
Jun 28 2020
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Sunday, 28 June 2020 at 16:37:23 UTC, Paul Backus wrote:

 On Sunday, 28 June 2020 at 15:33:48 UTC, Stanislav Blinov wrote:

 On Sunday, 28 June 2020 at 14:12:39 UTC, Paul Backus wrote:


 It sounds like what you are really trying to say here is that 
 input ranges and forward ranges should not use the same 
 interface, and that input ranges should instead implement 
 only `Option!T next()`.


 Not exactly. A given forward range may also implement such a 
 `next`. [...] We should differentiate by some other artifact. 
 I maintain that artifact should be copyability. Or we can keep 
 the distinction based on the presence of save(), and live 
 forever with the ambiguous semantics and the "don't use 
 original after copy unless you know what you're doing" 
 convention (yuck). Are there even other options besides 
 accepting UB?


 If input ranges are only required to implement next(), then the 
 distinction will be that forward ranges implement head()/tail() 
 and input ranges don't. (In this scenario, head() and tail() 
 will of course be required to return the same result if called 
 multiple times.)


Just as forward ranges may implement `next()`, some input ranges 
may want to be buffered, i.e.:

struct Wrapped
{
     private typeof(src.next()) buf; // Optional!ElementType
     private Input src; // Input is a `next()` input range

      property bool empty() const { return !buf; }
     T moveFront() { return buf.replace(T.init); }
     void popFront() { buf = src.next; }

     // no front, only moveFront
}

If we put a hard requirement on input ranges to *only* exist with 
the next() interface, this will shut down possible optimizations 
that buffering enables. We can't implement a buffering range in 
terms of the `next()` interface, a staggered fetch + advance is 
required. (next() may overwrite the internals of the buffer 
before client has a chance of reading the value).


 My example was a response to the specific claim that "an 
 immutable input range cannot exist." Specifically, it was a 
 counterexample demonstrating that the claim is false. As I'm 
 sure you can see, the claim is still false even if we agree 
 to use next() instead of tail() for input ranges, since 
 next() is obviously allowed to be impure.


 I can't see that. I don't see how you're falsifying that claim
 when in order to implement the range you need mutable state.




         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

 Ok, I'll show you. Here's the same range implemented using 
 next() instead of front() and tail():

             Optional!char next() const
             {
                 char c;
                 ssize_t nread = read(fd, cast(void*) &c, 1);


                                   ^^^^

                 if (nread == 0)
                     return no!char;
                 else if (nread == 1)
                     return some(c);
                 else
                     throw new Exception("Error reading file");
             }
         }
 Once again, `stdinByChar` is an input range, and it is 
 immutable. QED.


:) Fine. An input range can't exist without mutable state. Is 
that statement more agreeable for you?
Jun 29 2020
parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 29 June 2020 at 15:40:28 UTC, Stanislav Blinov wrote:

 We can't implement a buffering range in terms of the `next()` 
 interface, a staggered fetch + advance is required. (next() may 
 overwrite the internals of the buffer before client has a 
 chance of reading the value).


Are you referring to ranges like `byLine` that invalidate the 
previous front when popFront is called? If so, I don't see what 
difference it makes whether they're implemented with next() or 
some other interface.

If that's not what you're referring to, can you give an example?


 :) Fine. An input range can't exist without mutable state. Is 
 that statement more agreeable for you?


Yes, that's much more agreeable. :)

Personally, I am concerned here with how to make ranges work with 
D's `const` and `immutable` qualifiers. So the fact that there 
may be mutable state out there somewhere in libc/the kernel/the 
filesystem doesn't bother me.
Jun 29 2020
parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Monday, 29 June 2020 at 17:49:29 UTC, Paul Backus wrote:


 Are you referring to ranges like `byLine` that invalidate the 
 previous front when popFront is called?


Yes.


 If so, I don't see what difference it makes whether they're 
 implemented with next() or some other interface.


Efficiency. A given algorithm may need to stagger the range in 
order to preserve the head. To keep a `next()` interface and 
avoid buffer corruption, additional branch is needed for 
staggering (see [1]). This can be avoided by switching to 
front/popFront buffering where caller can explicitly announce 
they're done reading the element (not in the gist (yet)).

[1] 
https://gist.github.com/radcapricorn/d76d29c6df6fa822d7889e799937f39d#file-inputrange-d-L180


 Personally, I am concerned here with how to make ranges work 
 with D's `const` and `immutable` qualifiers. So the fact that 
 there may be mutable state out there somewhere in libc/the 
 kernel/the filesystem doesn't bother me.


I understand that. Forward ranges are good candidates for 
const/immutable. Input ranges though - I'd prefer to tread 
lightly there.
Jun 29 2020
prev sibling parent reply Mathias LANG <geod24 gmail.com> writes:
On Monday, 22 June 2020 at 21:46:51 UTC, Andrei Alexandrescu 
wrote:

 On 6/22/20 12:50 PM, Paul Backus wrote:


 IMHO the principled way to allow user-defined implicit 
 conversions is...to allow user-defined implicit conversions. 
 But iirc that's a can of worms Walter prefers not to open.


 What happens upon function calls is not an implicit conversion. 
 It's a forced type change.


Can you expand on this ? I've never heard of this before.
Jun 22 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/22/20 11:22 PM, Mathias LANG wrote:

 On Monday, 22 June 2020 at 21:46:51 UTC, Andrei Alexandrescu wrote:

 On 6/22/20 12:50 PM, Paul Backus wrote:


 IMHO the principled way to allow user-defined implicit conversions 
 is...to allow user-defined implicit conversions. But iirc that's a 
 can of worms Walter prefers not to open.


 What happens upon function calls is not an implicit conversion. It's a 
 forced type change.


 
 Can you expand on this ? I've never heard of this before.


https://run.dlang.io/is/KgeosK

Yah it's kinda surprising innit. For regular functions it's business as 
usual - exact match is attempted, conversions are considered etc. 
const(T[]) is convertible to const(T)[], nothing new about that.

For template functions however, although templates always do exact 
match, arrays surreptitiously change their type from const(T[]) to 
const(T)[], without a conversion in sight.

We need to offer that chance to other ranges if we want them to enjoy a 
similar treatment.
Jun 22 2020
parent reply Mathias LANG <geod24 gmail.com> writes:
On Tuesday, 23 June 2020 at 05:26:21 UTC, Andrei Alexandrescu 
wrote:

 On 6/22/20 11:22 PM, Mathias LANG wrote:

 On Monday, 22 June 2020 at 21:46:51 UTC, Andrei Alexandrescu 
 wrote:

 On 6/22/20 12:50 PM, Paul Backus wrote:


 IMHO the principled way to allow user-defined implicit 
 conversions is...to allow user-defined implicit conversions. 
 But iirc that's a can of worms Walter prefers not to open.


 What happens upon function calls is not an implicit 
 conversion. It's a forced type change.


 
 Can you expand on this ? I've never heard of this before.


 https://run.dlang.io/is/KgeosK

 Yah it's kinda surprising innit. For regular functions it's 
 business as usual - exact match is attempted, conversions are 
 considered etc. const(T[]) is convertible to const(T)[], 
 nothing new about that.

 For template functions however, although templates always do 
 exact match, arrays surreptitiously change their type from 
 const(T[]) to const(T)[], without a conversion in sight.

 We need to offer that chance to other ranges if we want them to 
 enjoy a similar treatment.


So it's not on function call, but it's about the type being 
deduced. There's quite a big difference here.

To me this makes perfect sense: This little peculiarity allows us 
to reduce the amount of template instantiations. E.g. the 
following code will only instantiate a single template:
```
void fun2(T)(const T[] a) { pragma(msg, T.stringof); }

void main() {
     immutable(int[]) a;
     fun2(a);
     int[] b;
     fun2(b);
}
```

In fact I'd love if we extend that to `auto` so the following 
code would work:
```
immutable(int[]) fun2() { return null; }

void main() {
     auto x = fun2();
     x = fun2(); // Currently `x` is `immutable(int[])` not 
`immutable(int)[]`
}
```
Jun 23 2020
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/23/20 5:45 AM, Mathias LANG wrote:

 On Tuesday, 23 June 2020 at 05:26:21 UTC, Andrei Alexandrescu wrote:

 https://run.dlang.io/is/KgeosK

 Yah it's kinda surprising innit. For regular functions it's business 
 as usual - exact match is attempted, conversions are considered etc. 
 const(T[]) is convertible to const(T)[], nothing new about that.

 For template functions however, although templates always do exact 
 match, arrays surreptitiously change their type from const(T[]) to 
 const(T)[], without a conversion in sight.

 We need to offer that chance to other ranges if we want them to enjoy 
 a similar treatment.


 
 So it's not on function call, but it's about the type being deduced. 
 There's quite a big difference here.


Yes, I agree. It works because of the implicit conversion.


 
 To me this makes perfect sense: This little peculiarity allows us to 
 reduce the amount of template instantiations. E.g. the following code 
 will only instantiate a single template:
 ```
 void fun2(T)(const T[] a) { pragma(msg, T.stringof); }
 
 void main() {
      immutable(int[]) a;
      fun2(a);
      int[] b;
      fun2(b);
 }
 ```


No, that would happen anyway, because in that case, T is stripped of the 
type modifier.

It's this case where it affects the result:

void fun(T)(T a) { pragma(msg, T.stringof); }

void main()
{
    const int[] a;
    const(int)[] b;
    fun(a);
    fun(b);
}

You only get one instantiation there.

The case makes complete sense, because the head is always a different 
memory location (a copy). What we lack is a way to make the head mutable 
of any type via a type modifier, so it's a "special case" in the sense 
that it's possible only on T[] and T*. If we change the special case to 
a general case, then it naturally makes things a lot more usable, and 
gets rid of a lot of the complaints about const ranges.


 
 In fact I'd love if we extend that to `auto` so the following code would 
 work:
 ```
 immutable(int[]) fun2() { return null; }
 
 void main() {
      auto x = fun2();
      x = fun2(); // Currently `x` is `immutable(int[])` not 
 `immutable(int)[]`
 }
 ```


That would be reasonable too, IMO.

-Steve
Jun 23 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/23/20 8:19 AM, Steven Schveighoffer wrote:

 On 6/23/20 5:45 AM, Mathias LANG wrote:

 On Tuesday, 23 June 2020 at 05:26:21 UTC, Andrei Alexandrescu wrote:

 https://run.dlang.io/is/KgeosK

 Yah it's kinda surprising innit. For regular functions it's business 
 as usual - exact match is attempted, conversions are considered etc. 
 const(T[]) is convertible to const(T)[], nothing new about that.

 For template functions however, although templates always do exact 
 match, arrays surreptitiously change their type from const(T[]) to 
 const(T)[], without a conversion in sight.

 We need to offer that chance to other ranges if we want them to enjoy 
 a similar treatment.


 So it's not on function call, but it's about the type being deduced. 
 There's quite a big difference here.


 
 Yes, I agree. It works because of the implicit conversion.


No, it doesn't work because of the implicit conversion, although 
implicit conversion is part of it. It works because there's a hack in 
the compiler that introduces the conversion from const(T[]) to 
const(T)[] automatically for template functions. It never happens for 
any other type - templates in D bind types "perfectly" otherwise. This 
is one exception.


 It's this case where it affects the result:
 
 void fun(T)(T a) { pragma(msg, T.stringof); }
 
 void main()
 {
     const int[] a;
     const(int)[] b;
     fun(a);
     fun(b);
 }
 
 You only get one instantiation there.


Yes, this is illustrating the hack.


 The case makes complete sense, because the head is always a different 
 memory location (a copy). What we lack is a way to make the head mutable 
 of any type via a type modifier, so it's a "special case" in the sense 
 that it's possible only on T[] and T*. If we change the special case to 
 a general case, then it naturally makes things a lot more usable, and 
 gets rid of a lot of the complaints about const ranges.


Correct, thanks. That's where T.opOnCall() would fit in - the function 
would effect a change of type when an object of type T is passed by 
value to a template function. It sounds awfully particular, but I'm 
encouraged by the fact that it has worked so well for so many years with 
arrays and pointers.
Jun 23 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/23/20 11:54 AM, Andrei Alexandrescu wrote:

 No, it doesn't work because of the implicit conversion, although 
 implicit conversion is part of it.


Whether the compiler hacks it by converting the type before invoking the 
type detection, or whether it hacks by adjusting the IFTI detection is 
not really important.

In the general case, though, there is no implicit conversion -- the 
conversion function must be used. So it would be important to call the 
conversion function before IFTI is invoked.

But another solution would be define implicit conversions that are 
preferred for IFTI/auto. Like an opCanonical or something. This I think 
sits better and is more consistent.

There are other cases aside from tail-const that could make IFTI more 
usable. For example: https://issues.dlang.org/show_bug.cgi?id=4998

This wouldn't be doable without some more help from the compiler during 
IFTI with possibly template constraints.

-Steve
Jun 23 2020
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 6/23/20 1:01 PM, Steven Schveighoffer wrote:

 On 6/23/20 11:54 AM, Andrei Alexandrescu wrote:

 No, it doesn't work because of the implicit conversion, although 
 implicit conversion is part of it.


 
 Whether the compiler hacks it by converting the type before invoking the 
 type detection, or whether it hacks by adjusting the IFTI detection is 
 not really important.


*nod*


 In the general case, though, there is no implicit conversion -- the 
 conversion function must be used. So it would be important to call the 
 conversion function before IFTI is invoked.
 
 But another solution would be define implicit conversions that are 
 preferred for IFTI/auto. Like an opCanonical or something. This I think 
 sits better and is more consistent.


Yah, that's what I meant with opOnCall. opOnAutoOrIFTI...


 There are other cases aside from tail-const that could make IFTI more 
 usable. For example: https://issues.dlang.org/show_bug.cgi?id=4998
 
 This wouldn't be doable without some more help from the compiler during 
 IFTI with possibly template constraints.


*nod*
Jun 23 2020
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/23/20 5:45 AM, Mathias LANG wrote:

 On Tuesday, 23 June 2020 at 05:26:21 UTC, Andrei Alexandrescu wrote:

 On 6/22/20 11:22 PM, Mathias LANG wrote:

 On Monday, 22 June 2020 at 21:46:51 UTC, Andrei Alexandrescu wrote:

 On 6/22/20 12:50 PM, Paul Backus wrote:


 IMHO the principled way to allow user-defined implicit conversions 
 is...to allow user-defined implicit conversions. But iirc that's a 
 can of worms Walter prefers not to open.


 What happens upon function calls is not an implicit conversion. It's 
 a forced type change.


 Can you expand on this ? I've never heard of this before.


 https://run.dlang.io/is/KgeosK

 Yah it's kinda surprising innit. For regular functions it's business 
 as usual - exact match is attempted, conversions are considered etc. 
 const(T[]) is convertible to const(T)[], nothing new about that.

 For template functions however, although templates always do exact 
 match, arrays surreptitiously change their type from const(T[]) to 
 const(T)[], without a conversion in sight.

 We need to offer that chance to other ranges if we want them to enjoy 
 a similar treatment.


 
 So it's not on function call, but it's about the type being deduced. 
 There's quite a big difference here.


I don't understand the difference. The change $qual(T[]) -> $qual(T)[] 
happens specifically when a template parameter type is deduced for a 
function call.


 To me this makes perfect sense: This little peculiarity allows us to 
 reduce the amount of template instantiations.


As I said before, I appreciate there's no shortage of people ready to 
explain things I've done back to me. In this case it's actually Walter's 
work. A long time ago, shortly after we introduced ranges and algorithms 
to Phobos, people complained (including in this forum) that there was no 
way to use algorithms with constant arrays. Walter and I talked about it 
for a while and he introduced this glorious hack, only for arrays and 
only for template function calls. Suddenly const arrays started to work, 
and nobody complained about the lack of generality of the hack. Until 
now I guess!


 E.g. the following code 
 will only instantiate a single template:
 ```
 void fun2(T)(const T[] a) { pragma(msg, T.stringof); }
 
 void main() {
      immutable(int[]) a;
      fun2(a);
      int[] b;
      fun2(b);
 }
 ```


This example is actually not illustrative. The first call involves a 
conversion from immutable(int[]) to immutable(int)[]. This happens even 
for non-templates. The second call involves a conversion from int[] to 
const(int[]), again a standard conversion. Neither call involves the 
hack discussed here, which is const(int[]) to const(int)[].


 In fact I'd love if we extend that to `auto` so the following code would 
 work:
 ```
 immutable(int[]) fun2() { return null; }
 
 void main() {
      auto x = fun2();
      x = fun2(); // Currently `x` is `immutable(int[])` not 
 `immutable(int)[]`
 }
 ```


That'd work but would not improve the problem of working with const ranges.
Jun 23 2020
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/23/20 11:50 AM, Andrei Alexandrescu wrote:

 E.g. the following code will only instantiate a single template:
 ```
 void fun2(T)(const T[] a) { pragma(msg, T.stringof); }

 void main() {
      immutable(int[]) a;
      fun2(a);
      int[] b;
      fun2(b);
 }
 ```


 
 This example is actually not illustrative. The first call involves a 
 conversion from immutable(int[]) to immutable(int)[].


Oops, I meant from immutable(int[]) to const(int[]).
Jun 23 2020
prev sibling next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Monday, 22 June 2020 at 14:33:53 UTC, Steven Schveighoffer 
wrote:

 [snip]

 FeepingCreature is right, we should try and create a head 
 mutable mechanism. I have envisioned it in the past as a 
 tail-modifier mechanism (e.g. tail-const).

 -Steve


I just read a thread from 2010 on tail const/head mutable. Kind 
of amazing that it is a difficult enough problem that it still 
hasn't been resolved.
Jun 22 2020
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 6/22/20 11:07 AM, jmh530 wrote:

 On Monday, 22 June 2020 at 14:33:53 UTC, Steven Schveighoffer wrote:

 [snip]

 FeepingCreature is right, we should try and create a head mutable 
 mechanism. I have envisioned it in the past as a tail-modifier 
 mechanism (e.g. tail-const).

 -Steve


 
 I just read a thread from 2010 on tail const/head mutable. Kind of 
 amazing that it is a difficult enough problem that it still hasn't been 
 resolved.


I don't think it's that difficult, it's that it has a rather large 
footprint. Qualifiers weigh heavily on the language.

I think going the lowering route with opCall() would be a much easier 
route, and also more expressive.
Jun 22 2020
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 6/22/20 10:33 AM, Steven Schveighoffer wrote:

 On 6/22/20 10:20 AM, Paul Backus wrote:

 On Monday, 22 June 2020 at 12:15:39 UTC, Steven Schveighoffer wrote:

 My question wasn't about how such a thing could be implemented, but 
 how it works with const ranges.

 foreach(x; someConstRange) I think wouldn't be possible. I think 
 you'd have to recurse:

 void process(const Range r)
 {
    subProcess(r.front);
    process(r.rest);
 }

 The point is to question the statement "so that we can have `const` 
 and `immutable` ranges".

 Sure, we could implement recursive versions of find, etc. I don't 
 know if that's worth it.


 Well, currently, range algorithms can't work with const ranges *at 
 all*, recursively or iteratively. So from a user perspective, this 
 would be a strict improvement on the status quo.


 
 Algorithms can work with const ranges -- as long as the range is an array:
 
 const int[] arr = [1, 2, 3, 4, 5];
 
 auto x = arr.find(3);
 
 assert(x == [3, 4, 5]);
 
 I think the better option is to focus on making it possible to duplicate 
 this possibility for generic ranges rather than implement a new and 
 awkward API.
 
 FeepingCreature is right, we should try and create a head mutable 
 mechanism. I have envisioned it in the past as a tail-modifier mechanism 
 (e.g. tail-const).


Oh, indeed. Arrays are special cased by the compiler - const(T[]) is 
silently converted to const(T)[] when calling a function.

At a point there was a discussion about allowing a similar conversion to 
be done automatically by the compiler - opCall(). So whenever you pass 
an object to a function, if the type defines opCall, it would be 
automatically invoked.
Jun 22 2020
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Jun 22, 2020 at 11:52:47AM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:
[...]

 At a point there was a discussion about allowing a similar conversion
 to be done automatically by the compiler - opCall(). So whenever you
 pass an object to a function, if the type defines opCall, it would be
 automatically invoked.


This conflicts with the function call operator, which is also called
.opCall.  I wouldn't want the function objects I pass around to be
"accidentally" invoked just because it defines .opCall!


T

-- 
In a world without fences, who needs Windows and Gates? -- Christian Surchi
Jun 22 2020
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 6/22/20 12:01 PM, H. S. Teoh wrote:

 On Mon, Jun 22, 2020 at 11:52:47AM -0400, Andrei Alexandrescu via
Digitalmars-d wrote:
 [...]

 At a point there was a discussion about allowing a similar conversion
 to be done automatically by the compiler - opCall(). So whenever you
 pass an object to a function, if the type defines opCall, it would be
 automatically invoked.


 
 This conflicts with the function call operator, which is also called
 .opCall.  I wouldn't want the function objects I pass around to be
 "accidentally" invoked just because it defines .opCall!


Oh, sorry. The name was different - possibly opOnCall.

All in all it's a matter of deciding on how important this problem is. 
(I think it is.)
Jun 22 2020
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/20 12:34 AM, H. S. Teoh wrote:

 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu via
Digitalmars-d wrote:
 [...]

 One good goal for std.v2020 would be to forego autodecoding
 throughout.


 [...]
 
 Another could be to fix up the range API -- i.e, reconsider the ugliness
 that is .save, now that D has copy ctors.


Awesome. So we have a lil list already.
Jun 20 2020
parent Jon Degenhardt <jond noreply.com> writes:
On Saturday, 20 June 2020 at 14:44:58 UTC, Andrei Alexandrescu 
wrote:

 On 6/20/20 12:34 AM, H. S. Teoh wrote:

 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu 
 via Digitalmars-d wrote:
 [...]

 One good goal for std.v2020 would be to forego autodecoding
 throughout.


 [...]
 
 Another could be to fix up the range API -- i.e, reconsider 
 the ugliness
 that is .save, now that D has copy ctors.


 Awesome. So we have a lil list already.


These two are high on my list as well.

Also high on my list is a hash table in the standard library that 
better manages memory for large numbers of entries. There are 
multi-second GC pauses when large numbers of entries are added to 
the current AAs. There was an attempt at addressing this a couple 
years ago, but the first try didn't pan out: 
https://github.com/dlang/druntime/pull/1929.

Might not require a new library version to address this.
Jun 21 2020
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, June 22, 2020 9:37:51 PM MDT Jonathan M Davis via Digitalmars-d 
wrote:

 Now, as far as basic input ranges go, having them be classes isn't
 necessarily a problem, since basic input ranges are essentially reference
 types by their very nature (or at least, they can't be value types). If they
 could have value semantics, they could be forward ranges. In fact, in some
 ways, it would be nice to require that basic input ranges be classes, since
 then it would be easy to introspect a basic input range vs a forward range,
 and it would ensure that basic input ranges have full-on reference
 semantics, but it also would result in heap allocations that are not
 currently necessary - particularly in cases where it would currently work
 to use a pseudo-reference type rather than full-on reference type. So,
 while it would be very nice to be able to require that basic input ranges
 be classes, I doubt that it's actually reasonable to do so. Either way, it
 shouldn't be a problem to allow basic input ranges to be classes. It's just
 with forward ranges that it's a problem, but struct wrappers should solve
 that problem.


On further reflection though, one advantage of disallowing classes as ranges
entirely (thus requiring a struct wrapper even for basic input ranges) would
be that we could require that the init value be a valid, empty range, which
we can't do as long as we allow classes. Still, the fact that basic input
ranges and forward ranges inherently have different copying semantics means
that they don't have quite the same problem with classes.

- Jonathan M Davis
Jun 22 2020
prev sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Thursday, 18 June 2020 at 15:55:06 UTC, jmh530 wrote:

 [snip]


Added bugzilla
https://issues.dlang.org/show_bug.cgi?id=20978
Jun 25 2020
prev sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
On Saturday, 13 June 2020 at 19:42:47 UTC, Andrei Alexandrescu 
wrote:

 I wonder how well this would work.


I've thought about this before and I think it has a lot of 
potential (back when we were first looking at dub, this was my 
preferred version scheme and everyone shot it down :( )

I believe if the new version just public imported the old version 
and then you add new/breaking functionality there, you could 
bring it all together with minimal duplication and maximum help 
from the compiler in making it work, including using two 
incompatible versions simultaneously because they'd have 
different mangles from different module names.

I'd like to see us try to play with it.
Jun 13 2020
prev sibling next sibling parent Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Saturday, 13 June 2020 at 03:33:14 UTC, Andrei Alexandrescu 
wrote:

 On 6/12/20 9:42 AM, 12345swordy wrote:

 On Friday, 12 June 2020 at 08:27:07 UTC, MoonlightSentinel 
 wrote:

 On Friday, 12 June 2020 at 00:24:41 UTC, 12345swordy wrote:

 Quite curious that I never see any attempt on it what so 
 ever.


 I assume you are talking about DMD's test runner 
 (https://github.com/dlang/dmd/blob/master/test/run.d).


 There are no concrete plan's AFAICT but I generally like the 
 idea to replace make for the other repos as well.


 
 I'm Sorry, I was referring to this:
 https://github.com/dlang/dmd/blob/master/src/build.d


 That should be killed with fire. I have seldom disliked a 
 program this much.


It's one of the best pieces of engineering in DMD :)
Shall we delete dmd then? :D
Jun 13 2020
prev sibling next sibling parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Saturday, 13 June 2020 at 03:33:14 UTC, Andrei Alexandrescu 
wrote:

 That should be killed with fire. I have seldom disliked a 
 program this much.


If anything should be killed with fire that should probably be 
the use of DDoc macros in dlang.org. It's a prime example of 
https://wiki.c2.com/?JobSecurity, but with the difference with 
that analogy that the only people needing that job security had 
already quit and now the whole thing is so messed up that no one 
can fix it without rewriting it from scratch.
Jun 13 2020
next sibling parent reply David Gileadi <gileadisNOSPM gmail.com> writes:
On 6/13/20 2:05 AM, Petar Kirov [ZombineDev] wrote:

 On Saturday, 13 June 2020 at 03:33:14 UTC, Andrei Alexandrescu wrote:

 That should be killed with fire. I have seldom disliked a program this 
 much.


 
 If anything should be killed with fire that should probably be the use 
 of DDoc macros in dlang.org. It's a prime example of 
 https://wiki.c2.com/?JobSecurity, but with the difference with that 
 analogy that the only people needing that job security had already quit 
 and now the whole thing is so messed up that no one can fix it without 
 rewriting it from scratch.


At one time I had ambitions to search and replace the documentation to 
use markdown syntax instead of Ddoc macros to the extent possible. I 
just haven't made time for it yet, sadly.

One in particular I'd like to replace is the silly $(P) macro on every 
single paragraph, but that would require another PR to Ddoc to do 
automatically create paragraphs instead of <BR>s. In my original 
"markdown" PR that got shot down and I didn't try to bring it back at 
the time.
Jun 13 2020
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Saturday, 13 June 2020 at 15:25:07 UTC, David Gileadi wrote:

 One in particular I'd like to replace is the silly $(P) macro 
 on every single paragraph


Fun fact: this is the key issue that caused me to actually create 
adrdox.

Lots of problems with ddoc but the horrible paragraph handling 
was the finishing blow.
Jun 13 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/13/20 11:33 AM, Adam D. Ruppe wrote:

 On Saturday, 13 June 2020 at 15:25:07 UTC, David Gileadi wrote:

 One in particular I'd like to replace is the silly $(P) macro on every 
 single paragraph


 
 Fun fact: this is the key issue that caused me to actually create adrdox.
 
 Lots of problems with ddoc but the horrible paragraph handling was the 
 finishing blow.


Yah that's awful. At a point either myself or someone else was working 
on automatically inserting a $(P ...) whenever a double newline was 
present. I think that was an easy fix.
Jun 13 2020
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Saturday, 13 June 2020 at 18:30:31 UTC, Andrei Alexandrescu 
wrote:

 Yah that's awful. At a point either myself or someone else was 
 working on automatically inserting a $(P ...) whenever a double 
 newline was present. I think that was an easy fix.


Indeed, and ddoc actually kinda does that for .d files, it just 
doesn't for .dd files. Weird rules.

But \n\n is not necessarily a paragraph break. Consider this:

/++

First paragraph

$(DIVC foo,
   bar
)
+/

That \n\n there is immediately followed by a div tag... and 
should not actually have a <p> before it. It isn't completely 
wrong but it isn't right semantically and can lead to spacing 
bugs in the resulting page (it is invalid html to have a div 
inside a p, so it will automatically close the <p> tag before 
opening the <div>).

So it is just a bit more complicated than it looks. My 
implementation to cover all the crazy edge cases I thought up is 
like 300 lines of code and it still isn't actually perfect (of 
course any time you have a 300 line buggy function there's the 
possibility that I just overcomplicated it and there's a much 
simpler, more general solution I missed).
Jun 15 2020
prev sibling next sibling parent Dennis <dkorpel gmail.com> writes:
On Saturday, 13 June 2020 at 09:05:46 UTC, Petar Kirov 
[ZombineDev] wrote:

 If anything should be killed with fire that should probably be 
 the use of DDoc macros in dlang.org.


Out of curiosity, are there any more details on that? Like a 
Bugzilla issue, forum post, PR discussion etc. This complaint is 
new to me.
Jun 13 2020
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/13/20 5:05 AM, Petar Kirov [ZombineDev] wrote:

 On Saturday, 13 June 2020 at 03:33:14 UTC, Andrei Alexandrescu wrote:

 That should be killed with fire. I have seldom disliked a program this 
 much.


 
 If anything should be killed with fire that should probably be the use 
 of DDoc macros in dlang.org. It's a prime example of 
 https://wiki.c2.com/?JobSecurity, but with the difference with that 
 analogy that the only people needing that job security had already quit 
 and now the whole thing is so messed up that no one can fix it without 
 rewriting it from scratch.


What's wrong with those? Also: don't forget that it's easy to define a 
battery of macros to translate to other formats. In addition to html 
there are translations to ebook, LaTeX, plain text, and one that I found 
interesting to define, which outputs the actual macros back. Or 
something like that. Oh, I found it: 
https://github.com/dlang/dlang.org/blob/master/verbatim.ddoc. Cool beans.

To translate to other formats automatically or semiautomatically you 
could start with such a battery of macros and define them appropriately.
Jun 13 2020
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Saturday, 13 June 2020 at 18:22:27 UTC, Andrei Alexandrescu 
wrote:

 What's wrong with those? Also: don't forget that it's easy to 
 define a battery of macros to translate to other formats.


This doesn't actually work since there's some characters with 
different meanings in those targets and ddoc's only answer to it 
is to introduce more and more macros for single characters.

It is a big hassle to actually use and gives no real world 
benefit since using html as an intermediate format is easier 
anyway!
Jun 15 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/15/20 12:58 PM, Adam D. Ruppe wrote:

 On Saturday, 13 June 2020 at 18:22:27 UTC, Andrei Alexandrescu wrote:

 What's wrong with those? Also: don't forget that it's easy to define a 
 battery of macros to translate to other formats.


 
 This doesn't actually work since there's some characters with different 
 meanings in those targets and ddoc's only answer to it is to introduce 
 more and more macros for single characters.


There is a way to translate individual characters before applying 
macros. I'm not sure of its limitations.
Jun 15 2020
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Tuesday, 16 June 2020 at 01:15:08 UTC, Andrei Alexandrescu 
wrote:

 There is a way to translate individual characters before 
 applying macros. I'm not sure of its limitations.


That only works on code blocks and inline `code`, actually. ddoc 
has a misfeature of "embedded html" that specifies this bad 
behavior in other contexts so I wasn't able to convince the 
language maintainers to change it globally.

https://dlang.org/spec/ddoc.html#embedded_html
Jun 15 2020
prev sibling parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Saturday, 13 June 2020 at 03:33:14 UTC, Andrei Alexandrescu 
wrote:

 That should be killed with fire. I have seldom disliked a 
 program this much.


I think the most accurate way to classify your message is as 
"shitposting", which according to the (5th) definition in the 
Urban Dictionary [1] is:
1: The failure to make a constructive post
2: The inability to add useful information to a forum
3: Worthless overly offensive generally racists posts written in 
a manner which aggravates others.
4: Nrom

Andrei, please use a more professional demeanor.

[1]: https://www.urbandictionary.com/define.php?term=shitposting
Jun 13 2020
next sibling parent reply Seb <seb wilzba.ch> writes:
On Saturday, 13 June 2020 at 09:24:52 UTC, Petar Kirov 
[ZombineDev] wrote:

 On Saturday, 13 June 2020 at 03:33:14 UTC, Andrei Alexandrescu 
 wrote:

 That should be killed with fire. I have seldom disliked a 
 program this much.


 I think the most accurate way to classify your message is as 
 "shitposting", which according to the (5th) definition in the 
 Urban Dictionary [1] is:
 1: The failure to make a constructive post
 2: The inability to add useful information to a forum
 3: Worthless overly offensive generally racists posts written 
 in a manner which aggravates others.
 4: Nrom

 Andrei, please use a more professional demeanor.

 [1]: https://www.urbandictionary.com/define.php?term=shitposting


I can't agree more. I miss the "original Andrei" :/
Jun 13 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/13/20 8:41 AM, Seb wrote:

 On Saturday, 13 June 2020 at 09:24:52 UTC, Petar Kirov [ZombineDev] wrote:

 On Saturday, 13 June 2020 at 03:33:14 UTC, Andrei Alexandrescu wrote:

 That should be killed with fire. I have seldom disliked a program 
 this much.


 I think the most accurate way to classify your message is as 
 "shitposting", which according to the (5th) definition in the Urban 
 Dictionary [1] is:
 1: The failure to make a constructive post
 2: The inability to add useful information to a forum
 3: Worthless overly offensive generally racists posts written in a 
 manner which aggravates others.
 4: Nrom

 Andrei, please use a more professional demeanor.

 [1]: https://www.urbandictionary.com/define.php?term=shitposting


 
 I can't agree more. I miss the "original Andrei" :/


The original Andrei, just like today's Andrei, has an appreciation for 
good engineering. I didn't feel the need to add to provide detail 
because (a) most regulars in this forum already knew what I was going to 
say, and (b) nobody save for a few would share my opinion.

But, I'll bite again, again to regret it.

The absolute minimum - and I repeat, absolute minimum, first thing, 
knee-jerk thing - that one would expect from a custom build program 
would be that it separates the data (cmdline options, file names, 
environment vars, everything that's supposed to change often and easily) 
from the code that manipulates them and deals with things like tracking 
dependencies, parsing things, etc. If all is implemented in a single 
file I'd expect the data first, rules next, details at the end. build.d 
almost seems to want to make a point to be nothing like that.

* Available rules (rootRules) nicely appear on line 40.

* Then, we have main() dealing with a lot of stuff. (Why?)

* The rules using makeRule are a confusing mix of code and data starting 
at line 249, after the reader has browsed through a bunch of 
implementation details. (An interesting side question would be what 
language features would make such things easier to write; e.g. a mixin 
string in conjunction with a DSL that implements a small subset of make 
may be nicer. But also becomes ironic - we started by going away from make.)

* Configuration file stuff is at lines 297-315.

* Hundreds of lines of rules follow. It is natural to ask oneself to 
what extent they improve on makefile (or other build tools') syntax. I 
mean is this something easy on the eyes? Did build.d attain its objective?

  582  /// BuildRule to generate man pages
    583  alias man = makeRule!((builder, rule) {
    584      alias genMan = methodInit!(BuildRule, (genManBuilder, 
genManRule) => genManBuilder
    585          .target(env["G"].buildPath("gen_man"))
    586          .sources([
    587              dmdRepo.buildPath("docs", "gen_man.d"),
    588              env["D"].buildPath("cli.d")])
    589          .command([
    590              env["HOST_DMD_RUN"],
    591              "-I" ~ srcDir,
    592              "-of" ~ genManRule.target]
    593              ~ flags["DFLAGS"]
    594              ~ genManRule.sources)
    595          .msg(genManRule.command.join(" "))
    596      );
    597
    598      const genManDir = env["GENERATED"].buildPath("docs", "man");
    599      alias dmdMan = methodInit!(BuildRule, (dmdManBuilder, 
dmdManRule) => dmdManBuilder
    600          .target(genManDir.buildPath("man1", "dmd.1"))
    601          .deps([genMan, directoryRule(dmdManRule.target.dirName)])
    602          .msg("(GEN_MAN) " ~ dmdManRule.target)
    603          .commandFunction(() {
    604              writeText(dmdManRule.target, 
genMan.target.execute.output);
    605          })
    606      );
    607      builder
    608      .name("man")
    609      .description("Generate and prepare man files")
    610      .deps([dmdMan].chain(
    611          "man1/dumpobj.1 man1/obj2asm.1 man5/dmd.conf.5".split
    612          .map!(e => methodInit!(BuildRule, (manFileBuilder, 
manFileRule) => manFileBuilder
    613              .target(genManDir.buildPath(e))
    614              .sources([dmdRepo.buildPath("docs", "man", e)])
    615              .deps([directoryRule(manFileRule.target.dirName)])
    616              .commandFunction(() => 
copyAndTouch(manFileRule.sources[0], manFileRule.target))
    617              .msg("copy '%s' to 
'%s'".format(manFileRule.sources[0], manFileRule.target))
    618          ))
    619      ).array);
    620  });

* The source files are hardcoded somewhere starting at line 1160. I wish 
I was kidding.

* A lot more support code follows, which is fine for a single-file build 
tool. I wish this were a two-files affair a la build.d and buildinfo.d, 
or if single file have an obvious separation:

/****************************************************************
     IMPLEMENTATION FOLLOWS
****************************************************************/
Jun 13 2020
next sibling parent reply Dennis <dkorpel gmail.com> writes:
On Saturday, 13 June 2020 at 18:56:55 UTC, Andrei Alexandrescu 
wrote:

 I didn't feel the need to add to provide detail because (a) 
 most regulars in this forum already knew what I was going to 
 say, and (b) nobody save for a few would share my opinion.

 But, I'll bite again, again to regret it.


I, for one, thought it was an interesting little write-up. Didn't 
expect it, my guess was that you were against the idea of using D 
to build something instead of a DSL, not the implementation.
Jun 13 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/13/20 6:52 PM, Dennis wrote:

 On Saturday, 13 June 2020 at 18:56:55 UTC, Andrei Alexandrescu wrote:

 I didn't feel the need to add to provide detail because (a) most 
 regulars in this forum already knew what I was going to say, and (b) 
 nobody save for a few would share my opinion.

 But, I'll bite again, again to regret it.


 
 I, for one, thought it was an interesting little write-up. Didn't expect 
 it, my guess was that you were against the idea of using D to build 
 something instead of a DSL, not the implementation.


I should add - the fact that dmd needs to be installed in order to build 
dmd is the proverbial insult added to the injury. Of course that ruined 
the carefully constructed AUTO_BOOTSTRAP option that allows building dmd 
on a fresh system.

I should also add - unless I'm looking at the wrong version, the old 
posix.mak has 654 lines. build.d has 1932 lines. But build.d also 
supplants the Windows 32/64 makefiles (589/57 lines), so the size is not 
way bigger. But that begs the question - given its liabilities, by what 
metric is build.d an improvement?
Jun 13 2020
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Saturday, 13 June 2020 at 23:53:25 UTC, Andrei Alexandrescu 
wrote:

 On 6/13/20 6:52 PM, Dennis wrote:

 On Saturday, 13 June 2020 at 18:56:55 UTC, Andrei Alexandrescu 
 wrote:

 I didn't feel the need to add to provide detail because (a) 
 most regulars in this forum already knew what I was going to 
 say, and (b) nobody save for a few would share my opinion.

 But, I'll bite again, again to regret it.


 
 I, for one, thought it was an interesting little write-up. 
 Didn't expect it, my guess was that you were against the idea 
 of using D to build something instead of a DSL, not the 
 implementation.


 I should add - the fact that dmd needs to be installed in order 
 to build dmd is the proverbial insult added to the injury. Of 
 course that ruined the carefully constructed AUTO_BOOTSTRAP 
 option that allows building dmd on a fresh system.

 I should also add - unless I'm looking at the wrong version, 
 the old posix.mak has 654 lines. build.d has 1932 lines. But 
 build.d also supplants the Windows 32/64 makefiles (589/57 
 lines), so the size is not way bigger. But that begs the 
 question - given its liabilities, by what metric is build.d an 
 improvement?


You used to have to update diffrent makefiles when adding a new 
file the build

Have you used the windows makefiles?
They didn't work without modification.

I do agree that build.d takes to long to build because of all the 
phobos-ness in it.
But if I want to debug it I can, at least.

whereas with those makefiles which have to work with digital mars 
make.
it's way harder.
Jun 13 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/13/20 8:00 PM, Stefan Koch wrote:

 On Saturday, 13 June 2020 at 23:53:25 UTC, Andrei Alexandrescu wrote:

 On 6/13/20 6:52 PM, Dennis wrote:

 On Saturday, 13 June 2020 at 18:56:55 UTC, Andrei Alexandrescu wrote:

 I didn't feel the need to add to provide detail because (a) most 
 regulars in this forum already knew what I was going to say, and (b) 
 nobody save for a few would share my opinion.

 But, I'll bite again, again to regret it.


 I, for one, thought it was an interesting little write-up. Didn't 
 expect it, my guess was that you were against the idea of using D to 
 build something instead of a DSL, not the implementation.


 I should add - the fact that dmd needs to be installed in order to 
 build dmd is the proverbial insult added to the injury. Of course that 
 ruined the carefully constructed AUTO_BOOTSTRAP option that allows 
 building dmd on a fresh system.

 I should also add - unless I'm looking at the wrong version, the old 
 posix.mak has 654 lines. build.d has 1932 lines. But build.d also 
 supplants the Windows 32/64 makefiles (589/57 lines), so the size is 
 not way bigger. But that begs the question - given its liabilities, by 
 what metric is build.d an improvement?


 
 You used to have to update diffrent makefiles when adding a new file the 
 build


Awesome, now I only need to do surgery on build.d in an obscure function 
at line 1160 :o).

I'm not saying what was there was better. I'm just asking why there's so 
little progress.


 Have you used the windows makefiles?
 They didn't work without modification.
 
 I do agree that build.d takes to long to build because of all the 
 phobos-ness in it.
 But if I want to debug it I can, at least.
 
 whereas with those makefiles which have to work with digital mars make.
 it's way harder.


I've often said gmake should be used for Windows to build dmd. Now we 
require dmd to build dmd...

And this time is on us. We can't blame the arcane make syntax. We got to 
define it all, and defined it to be even more arcane.

Where. Is. The. Progress.
Jun 13 2020
next sibling parent reply Dennis <dkorpel gmail.com> writes:
On Sunday, 14 June 2020 at 00:22:29 UTC, Andrei Alexandrescu 
wrote:

 Now we require dmd to build dmd...


We do anyway ever since the code base got converted from C++ to D.
Jun 13 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/13/20 8:59 PM, Dennis wrote:

 On Sunday, 14 June 2020 at 00:22:29 UTC, Andrei Alexandrescu wrote:

 Now we require dmd to build dmd...


 
 We do anyway ever since the code base got converted from C++ to D.


As I mentioned, that was done carefully and beautifully by an 
AUTO_BOOTSTRAP option (thanks Martin Nowak). Now build.d is the first 
thing that needs to run, so the entire AUTO_BOOTSTRAP thing is broken.

Did I mention I appreciate good engineering?
Jun 13 2020
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Sunday, 14 June 2020 at 01:26:02 UTC, Andrei Alexandrescu 
wrote:

 On 6/13/20 8:59 PM, Dennis wrote:

 On Sunday, 14 June 2020 at 00:22:29 UTC, Andrei Alexandrescu 
 wrote:

 Now we require dmd to build dmd...


 
 We do anyway ever since the code base got converted from C++ 
 to D.


 As I mentioned, that was done carefully and beautifully by an 
 AUTO_BOOTSTRAP option (thanks Martin Nowak). Now build.d is the 
 first thing that needs to run, so the entire AUTO_BOOTSTRAP 
 thing is broken.

 Did I mention I appreciate good engineering?


make -f posix.mak AUTO_BOOTSTRAP is still a thing.
Jun 13 2020
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/13/20 9:30 PM, Stefan Koch wrote:

 On Sunday, 14 June 2020 at 01:26:02 UTC, Andrei Alexandrescu wrote:

 On 6/13/20 8:59 PM, Dennis wrote:

 On Sunday, 14 June 2020 at 00:22:29 UTC, Andrei Alexandrescu wrote:

 Now we require dmd to build dmd...


 We do anyway ever since the code base got converted from C++ to D.


 As I mentioned, that was done carefully and beautifully by an 
 AUTO_BOOTSTRAP option (thanks Martin Nowak). Now build.d is the first 
 thing that needs to run, so the entire AUTO_BOOTSTRAP thing is broken.

 Did I mention I appreciate good engineering?


 
 make -f posix.mak AUTO_BOOTSTRAP is still a thing.


Not when I tried it: 
https://forum.dlang.org/post/r5e8uj$i1b$1 digitalmars.com
Jun 13 2020
prev sibling parent reply Max Samukha <maxsamukha gmail.com> writes:
On Sunday, 14 June 2020 at 00:22:29 UTC, Andrei Alexandrescu 
wrote:


 Where. Is. The. Progress.


Progress would be to fix the makefiles so that they are usable 
out-of-the-box with the newer versions of VS. This is abysmal:

VCDIR=\Program Files (x86)\Microsoft Visual Studio 10.0\VC
SDKDIR=\Program Files (x86)\Microsoft SDKs\Windows\v7.0A

...

CC=$(VCDIR)\bin\amd64\cl
LD=$(VCDIR)\bin\amd64\link
AR=$(VCDIR)\bin\amd64\lib
Jun 13 2020
parent Seb <seb wilzba.ch> writes:
On Sunday, 14 June 2020 at 06:07:40 UTC, Max Samukha wrote:

 On Sunday, 14 June 2020 at 00:22:29 UTC, Andrei Alexandrescu 
 wrote:


 Where. Is. The. Progress.


 Progress would be to fix the makefiles so that they are usable 
 out-of-the-box with the newer versions of VS. This is abysmal:

 VCDIR=\Program Files (x86)\Microsoft Visual Studio 10.0\VC
 SDKDIR=\Program Files (x86)\Microsoft SDKs\Windows\v7.0A

 ...

 CC=$(VCDIR)\bin\amd64\cl
 LD=$(VCDIR)\bin\amd64\link
 AR=$(VCDIR)\bin\amd64\lib


For DMD, the Makefiles are only there for legacy CI reasons or 
AUTO_BOOTSTRAP and they might even be fully removed soonish.

So at least to build DMD, everything should work out of the box 
on any platform when executing `./src/build.d` (or `rdmd 
src\build.d` on Windows) and there should be no need to fight 
with the hard-coded paths to Visual Studio 10.
If not, please open issues on Bugzilla (and/or report here).
Jun 14 2020
prev sibling next sibling parent Avrina <avrina12309412342 gmail.com> writes:
On Saturday, 13 June 2020 at 18:56:55 UTC, Andrei Alexandrescu 
wrote:

 * Hundreds of lines of rules follow. It is natural to ask 
 oneself to what extent they improve on makefile (or other build 
 tools') syntax. I mean is this something easy on the eyes? Did 
 build.d attain its objective?

  582  /// BuildRule to generate man pages
    583  alias man = makeRule!((builder, rule) {
    584      alias genMan = methodInit!(BuildRule, 
 (genManBuilder, genManRule) => genManBuilder
    585          .target(env["G"].buildPath("gen_man"))
    586          .sources([
    587              dmdRepo.buildPath("docs", "gen_man.d"),
    588              env["D"].buildPath("cli.d")])
    589          .command([
    590              env["HOST_DMD_RUN"],
    591              "-I" ~ srcDir,
    592              "-of" ~ genManRule.target]
    593              ~ flags["DFLAGS"]
    594              ~ genManRule.sources)
    595          .msg(genManRule.command.join(" "))
    596      );
    597
    598      const genManDir = 
 env["GENERATED"].buildPath("docs", "man");
    599      alias dmdMan = methodInit!(BuildRule, 
 (dmdManBuilder, dmdManRule) => dmdManBuilder
    600          .target(genManDir.buildPath("man1", "dmd.1"))
    601          .deps([genMan, 
 directoryRule(dmdManRule.target.dirName)])
    602          .msg("(GEN_MAN) " ~ dmdManRule.target)
    603          .commandFunction(() {
    604              writeText(dmdManRule.target, 
 genMan.target.execute.output);
    605          })
    606      );
    607      builder
    608      .name("man")
    609      .description("Generate and prepare man files")
    610      .deps([dmdMan].chain(
    611          "man1/dumpobj.1 man1/obj2asm.1 
 man5/dmd.conf.5".split
    612          .map!(e => methodInit!(BuildRule, 
 (manFileBuilder, manFileRule) => manFileBuilder
    613              .target(genManDir.buildPath(e))
    614              .sources([dmdRepo.buildPath("docs", "man", 
 e)])
    615              
 .deps([directoryRule(manFileRule.target.dirName)])
    616              .commandFunction(() => 
 copyAndTouch(manFileRule.sources[0], manFileRule.target))
    617              .msg("copy '%s' to 
 '%s'".format(manFileRule.sources[0], manFileRule.target))
    618          ))
    619      ).array);
    620  });


I agree, "chaining" is touted as one of the benefits of D, but 
whenever I see people using them they tend to be these types of 
monstrosities with many inefficiencies. The developers tend to 
not even know what's going on as most of the logic is hidden.
Jun 13 2020
prev sibling parent reply Seb <seb wilzba.ch> writes:
On Saturday, 13 June 2020 at 18:56:55 UTC, Andrei Alexandrescu 
wrote:

 The original Andrei, just like today's Andrei, has an 
 appreciation for good engineering. I didn't feel the need to 
 add to provide detail because (a) most regulars in this forum 
 already knew what I was going to say, and (b) nobody save for a 
 few would share my opinion.

 But, I'll bite again, again to regret it.


I'll reply with similar regrets, because I agree that this 
discussion isn't helping anyone.
tl;dr: I agree with you that build.d isn't the ideal engineering 
piece that D was supposed to enable, but it did solve a few real 
problems and using D without dependency was really the only 
option to move forward:

In Detail:

1) DigitalMars Make was a massive real problem (and still is for 
Druntime+Phobos)
2) Duplication into many similar Makefiles with mismatching 
definitions (posix.mak, win32.mak, win64.mak, osmodel.mak, ...)
3) There were many things we couldn't model properly in Make 
(e.g. version detection shell script, documentation build, ...)
4) Especially the Windows Makefiles were missing a lot of target 
and options (e.g. no LTO/PGO builds, no library builds, ...)

https://github.com/dlang/dmd/blob/v2.080.0/src/posix.mak
https://github.com/dlang/dmd/blob/v2.080.0/src/win32.mak
https://github.com/dlang/dmd/blob/v2.080.0/src/win64.mak

5) You and Walter repeatedly struck down all attempts to use any 
other build tool (like e.g. reggae for Phobos - 
https://github.com/dlang/phobos/pull/4194)
6) The Windows Makefiles looked liked this:

---
$G/nteh.obj : $C\rtlsym.h $C\nteh.c
	$(CC) -c -o$  $(MFLAGS) $C\nteh

$G/os.obj : $C\os.c
	$(CC) -c -o$  $(MFLAGS) $C\os

$G/out.obj : $C\out.c
	$(CC) -c -o$  $(MFLAGS) $C\out

$G/outbuf.obj : $C\outbuf.h $C\outbuf.c
	$(CC) -c -o$  $(MFLAGS) $C\outbuf

$G/pdata.obj : $C\pdata.c
	$(CC) -c -o$  $(MFLAGS) $C\pdata

$G/ph2.obj : $C\ph2.c
	$(CC) -c -o$  $(MFLAGS) $C\ph2
---

7) Also, complexity in general increased over time as C++ header 
generation or CXX compatibility tests got added. There are a 
couple of other new features that are quite helpful for CIs (like 
e.g. the very detailed failure message on build failures).

In build.d there definitely are a few relicts from the Makefiles 
times as it was a lot easier to do a step-by-step migration of 
the targets than migrating everything in one bulk, so I'm not 
denying that the build script source code could be refactored and 
improved, but so could be almost any code in the D ecosystem.

Lastly, if you really want to know why build.d grew from 250 
lines initially to its current size, the answer is rather simple. 
It's because of many, many shortcomings and bugs in Druntime and 
Phobos.
As a prime example I encourage you to read through is:

https://github.com/dlang/dmd/pull/10611 (GitHub hides most of the 
discussion by default)


 * Then, we have main() dealing with a lot of stuff. (Why?)


Not sure, what's a "a lot of stuff" for you, but that's the main 
method as of now:

---
int main(string[] args)
{
     try
     {
         runMain(args);
         return 0;
     }
     catch (BuildException e)
     {
         // Ensure paths are relative to the root directory
         // s.t. error messages are clickable in most IDE's
         writeln(e.msg.replace(buildPath("dmd", ""), 
buildPath("src", "dmd", "")));
         return 1;
     }
}
---

To be fair, runMain does more things (gettings arguments + 
starting thread pool) and of course, the showHelp() could have 
easily been moved outside, but I really don't see how this is a 
critical problem for D?
Especially considering how big main() is at "your" rdmd:

https://github.com/dlang/tools/blob/master/rdmd.d#L62


 * The rules using makeRule are a confusing mix of code and data 
 starting at line 249, after the reader has browsed through a 
 bunch of implementation details.


1) Those are all rules. The implementation details are the 
rules...
2) The "reader" has a nice overview of all rules and targets at 
the beginning (before implementation)
3)) A typical "reader" is much more likely to execute ./build.d 
-h (or --help) than reading the source


 An interesting side question  would be what language features
 would make such things easier to write; e.g. a mixin string
 in conjunction with a DSL that implements a small subset of 
 make may be nicer.


Maybe a standard library with isn't riddled with hundreds of bugs 
and old/outdated/unusable code?


 But also becomes ironic - we started by going away from make.)


It's not ironic. WE CAN'T USE MAKE ON WINDOWS.


 * Hundreds of lines of rules follow.


Most of these rules are to support ugly things like e.g. the 
complicated ddoc or to workaround shortcomings in the standard 
library (e.g. no safe cross-platform scheduler).


 Did build.d attain its objective?


Absolutely. One big example is that people can now build DMD on 
Windows (this was _very_ hard/near to impossible before).


 * The source files are hardcoded somewhere starting at line 
 1160. I wish I was kidding.


build.d used globbing initially. It was removed, because Walter 
explicitly requested hard-coding.
I can't find the PR with the request at the moment, but here's 
one of the PRs which removed the globbing in favor of hard-coding:

https://github.com/dlang/dmd/pull/10212


 * A lot more support code follows, which is fine for a 
 single-file build tool. I wish this were a two-files affair a 
 la build.d and buildinfo.d, or if single file have an obvious 
 separation:


While we could have done a separation for rdmd which you struck 
done a while ago [1] which btw lead to the only two people 
interested in contributing to rdmd loosing their interest.

Now, to answer your question fully I have to add that one of 
those two people (Jonathan Marler) has since then decided to 
create a better rdmd: rund [2]. Rund fixes one of the main 
shortcomings of rdmd (the need to compile a file twice with dmd 
and thus having an almost doubled run time) and adds a couple of 
other handy features like being able to specify execution flags, 
e.g.

---

//!importPath <path>
---

If rdmd would have such a feature, we could split build.d into 
multiple files. Unfortunately, it doesn't and thus it must remain 
a single file as `./build.d` needs to work out of the box.

[1] https://github.com/dlang/tools/pull/344
[2] https://github.com/dragon-lang/rund


 or if single file have an obvious separation:


The separation is already there [1] , but I guess one could make 
it more obious.

[1] https://github.com/dlang/dmd/blob/master/src/build.d#L768


 I've often said gmake should be used for Windows to build dmd. 
 Now we require dmd to build dmd...


Yes, how else is one going to build D code since 2.067?


 Of course that ruined the carefully constructed AUTO_BOOTSTRAP 
 option that allows building dmd on a fresh system.


Nope. In fact, AUTO_BOOTSTRAP is still being used by some of our 
own CIs, e.g. Buildkite still does

```
make -f posix.mak AUTO_BOOTSTRAP=1
```

Some other do this to get an active D compiler in your 
environment:

```
source $(curl https://dlang.org/install.sh | bash -s install -a)
```


 Not when I tried it: 
 https://forum.dlang.org/post/r5e8uj$i1b$1 digitalmars.com


To be fair that's not a fault of build.d. You aren't able to 
compile anything with an outdated compiler on OSX. For details 
see some of the responses to your thread:

https://forum.dlang.org/post/rimiuawxnzobcgfuwxez forum.dlang.org
https://forum.dlang.org/post/zvsjvhvgprhgacybvctb forum.dlang.org
Jun 14 2020
next sibling parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Sunday, 14 June 2020 at 16:07:16 UTC, Seb wrote:

 On Saturday, 13 June 2020 at 18:56:55 UTC, Andrei Alexandrescu 
 wrote:

 The original Andrei, just like today's Andrei, has an 
 appreciation for good engineering. I didn't feel the need to 
 add to provide detail because (a) most regulars in this forum 
 already knew what I was going to say, and (b) nobody save for 
 a few would share my opinion.

 But, I'll bite again, again to regret it.


 I'll reply with similar regrets, because I agree that this 
 discussion isn't helping anyone.


It's helping me.

[big snip of clarifying context and concise refutations of many 
earlier assertions]

Thanks Seb, for the context, the refutations, and for the 
references to alternative tools (rund, reggae).
Jun 14 2020
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/14/20 1:05 PM, Bruce Carneal wrote:

 On Sunday, 14 June 2020 at 16:07:16 UTC, Seb wrote:

 On Saturday, 13 June 2020 at 18:56:55 UTC, Andrei Alexandrescu wrote:

 The original Andrei, just like today's Andrei, has an appreciation 
 for good engineering. I didn't feel the need to add to provide detail 
 because (a) most regulars in this forum already knew what I was going 
 to say, and (b) nobody save for a few would share my opinion.

 But, I'll bite again, again to regret it.


 I'll reply with similar regrets, because I agree that this discussion 
 isn't helping anyone.


 
 It's helping me.
 
 [big snip of clarifying context and concise refutations of many earlier 
 assertions]
 
 Thanks Seb, for the context, the refutations, and for the references to 
 alternative tools (rund, reggae).


Most of said refutations are reducible to simple misunderstandings. A 
few are due to my oversights. A few are Seb's. My core point stays: 
build.d is not Good Work(tm), and can't be talked into it. It must be 
worked into it.

I don't want the makefiles back. I want Good Work, which build.d has a 
good way to go toward. A good part of my critique can be addressed by 
refactoring build.d in the ways I suggested. Another part of the 
critique would be achieved by a pass through its design, with an eye for 
making dependency syntax tolerable.
Jun 14 2020
parent Bruce Carneal <bcarneal gmail.com> writes:
On Sunday, 14 June 2020 at 17:40:04 UTC, Andrei Alexandrescu 
wrote:

 On 6/14/20 1:05 PM, Bruce Carneal wrote:

 On Sunday, 14 June 2020 at 16:07:16 UTC, Seb wrote:

 On Saturday, 13 June 2020 at 18:56:55 UTC, Andrei


 [big snip of clarifying context and concise refutations of 
 many earlier assertions]
 
 Thanks Seb, for the context, the refutations, and for the 
 references to alternative tools (rund, reggae).


 Most of said refutations are reducible to simple 
 misunderstandings. A few are due to my oversights. A few are 
 Seb's. My core point stays: build.d is not Good Work(tm), and 
 can't be talked into it. It must be worked into it.

 I don't want the makefiles back. I want Good Work, which 
 build.d has a good way to go toward. A good part of my critique 
 can be addressed by refactoring build.d in the ways I 
 suggested. Another part of the critique would be achieved by a 
 pass through its design, with an eye for making dependency 
 syntax tolerable.


I'm not a domain expert on this but from my earlier sight-reading 
of the code it sure looks like a rework/refactor/rewrite 
candidate to me as well.  More importantly Seb, a domain expert, 
appears to agree with you on this.

Such an agreement would make this a part of the well recognized 
bigger challenge: engaging qualified programmers from a 
mostly-volunteer pool to address known problems.
Jun 14 2020
prev sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Sunday, 14 June 2020 at 16:07:16 UTC, Seb wrote:

 While we could have done a separation for rdmd which you struck 
 done a while ago [1] which btw lead to the only two people 
 interested in contributing to rdmd loosing their interest.

 ...

 [1] https://github.com/dlang/tools/pull/344


It's a bit tangential to the main thrust of discussion here, but 
I'd just like it on the record that the decision on that PR had 
nothing to do with why I didn't continue to contribute to rdmd.  
I just had a lot of other things keeping me busy (and still do).

Andrei and I had a very constructive conversation at DConf later 
that year where he suggested, and I agreed, that the way forward 
with rdmd's issues was not to refactor rdmd but to bring more of 
its functionality inside the D frontend (of which the main things 
would presumably be the search for the files to compile, and the 
running of the compiled executable).

I don't know what if any progress has been made in that respect 
but it still sounds the most reasonable way to clean things up 
and avoid duplication of effort between the codebases.

BTW, I think it is important to listen carefully to Andrei's 
broader feedback here.  A lot of the reason why I got hands on 
with rdmd was because I was seeing a lot of what I felt were 
problematic development decisions, and I wanted to clean up the 
codebase and put in place much more rigorous tests to help avoid 
those problems in future.

I know it's not easy to hear feedback like this from someone who 
isn't hands-on at the coalface alongside you, but it really feels 
like far too often work gets accepted because "something must be 
done!" and in order to encourage the person who is stepping up to 
do the work, rather than asking the harder question of, is it 
being done the right way, and is that work really producing the 
right long-term outcome.
Jun 23 2020
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/13/20 5:24 AM, Petar Kirov [ZombineDev] wrote:

 On Saturday, 13 June 2020 at 03:33:14 UTC, Andrei Alexandrescu wrote:

 That should be killed with fire. I have seldom disliked a program this 
 much.


 
 I think the most accurate way to classify your message is as 
 "shitposting", which according to the (5th) definition in the Urban 
 Dictionary [1] is:
 1: The failure to make a constructive post
 2: The inability to add useful information to a forum
 3: Worthless overly offensive generally racists posts written in a 
 manner which aggravates others.
 4: Nrom
 
 Andrei, please use a more professional demeanor.
 
 [1]: https://www.urbandictionary.com/define.php?term=shitposting


Nice going. Particularly lovely is the weaseling of loaded words that 
make it all look more righteous.
Jun 13 2020