www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Is run.d going to be expand for runtime and the phobos library?

reply 12345swordy <alexanderheistermann gmail.com> writes:
Quite curious that I never see any attempt on it what so ever.
Jun 11
next sibling parent reply Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Friday, 12 June 2020 at 00:24:41 UTC, 12345swordy wrote:
 Quite curious that I never see any attempt on it what so ever.
You leave me guessing what run.d is, and so I have no idea what you are curious about. — Bastiaan.
Jun 12
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Friday, 12 June 2020 at 07:38:22 UTC, Bastiaan Veelo wrote:
 On Friday, 12 June 2020 at 00:24:41 UTC, 12345swordy wrote:
 Quite curious that I never see any attempt on it what so ever.
You leave me guessing what run.d is, and so I have no idea what you are curious about. — Bastiaan.
(This is a message from a day ago that I forgot to send earlier.) TL;DR See the Readme file here: https://github.com/dlang/dmd/tree/v2.092.1/test and run.d itself: https://github.com/dlang/dmd/blob/v2.092.1/test/run.d. Also you can check: https://github.com/dlang/dmd/blob/v2.092.1/src/build.d --- The basic idea is that we're replacing the ugly makefile mess with a build and test system written in pure D. The advantages going for a pure D solution are numerous: - Developers and package maintainers need less external dependencies to build and test the code - There are several Make(1) implementations (e.g. GNUmake, BSDmake, DMmake [1]) each with different feature sets and limitations. This was especially a problem on Windows, as DMmake has much more limited feature set then GNUmake, which prevented us from having a single cross-platform Makefile and instead we had to maintain posix.mak, win32.mak and win64.mak - AFAIK, installing a GNUmake on Windows involves installing a whole Posix emulation/compatibility environment like cygwin, msys or msys2, which brings it's own rats nest of problems - Even if Make is not a problem, it is still dependent on Posix compatible shell, so that's one more variable in the equation - By implementing the full build system in D we have much more flexibility and it's easier to troubleshoot problems. Also we can implement features which are unlikely to ever become part of make, but would be useful for us. - For example, it's we can tune the scheduling of the test runner relatively easily, while with Make we he no control over that (it's just doing topological sorting, without any semantic understanding of what each target represents). - And quite a few other benefits... [1]: https://digitalmars.com/ctg/make.html
Jun 13
parent reply kinke <noone nowhere.com> writes:
On Saturday, 13 June 2020 at 08:58:56 UTC, Petar Kirov 
[ZombineDev] wrote:
 - AFAIK, installing a GNUmake on Windows involves installing a 
 whole Posix emulation/compatibility environment like cygwin, 
 msys or msys2, which brings it's own rats nest of problems
 - Even if Make is not a problem, it is still dependent on Posix 
 compatible shell, so that's one more variable in the equation
Side note: LDC has been using GNUmake on Windows for years, incl. extending some upstream Makefiles to work with and test Windows targets too. While GNUmake itself is surprisingly portable (source comes with a Visual Studio project working out of the box, resulting in a little 330 KB Win64 stand-alone executable), the tests require bash and the usual little GNU tools. Fortunately, a normal git installation on Windows provides the expected GNU environment, incl. bash.
Jun 13
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/13/2020 6:44 AM, kinke wrote:
 Side note: LDC has been using GNUmake on Windows for years, incl. extending
some 
 upstream Makefiles to work with and test Windows targets too.
 While GNUmake itself is surprisingly portable (source comes with a Visual
Studio 
 project working out of the box, resulting in a little 330 KB Win64 stand-alone 
 executable),
DM make is 52K :-)
Jun 13
parent reply Avrina <avrina12309412342 gmail.com> writes:
On Sunday, 14 June 2020 at 04:26:00 UTC, Walter Bright wrote:
 On 6/13/2020 6:44 AM, kinke wrote:
 Side note: LDC has been using GNUmake on Windows for years, 
 incl. extending some upstream Makefiles to work with and test 
 Windows targets too.
 While GNUmake itself is surprisingly portable (source comes 
 with a Visual Studio project working out of the box, resulting 
 in a little 330 KB Win64 stand-alone executable),
DM make is 52K :-)
Size doesn't matter if it doesn't work.
Jun 14
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/14/2020 9:18 AM, Avrina wrote:
 Size doesn't matter if it doesn't work.
I use it all the time, it works fine. It does lack the features that make makefiles incomprehensible, though.
Jun 20
next sibling parent Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Saturday, 20 June 2020 at 20:13:26 UTC, Walter Bright wrote:
 On 6/14/2020 9:18 AM, Avrina wrote:
 Size doesn't matter if it doesn't work.
I use it all the time, it works fine.
This one is preventing me from using optlink: https://issues.dlang.org/show_bug.cgi?id=15213 —Bastiaan
Jun 20
prev sibling parent Avrina <avrina12309412342 gmail.com> writes:
On Saturday, 20 June 2020 at 20:13:26 UTC, Walter Bright wrote:
 On 6/14/2020 9:18 AM, Avrina wrote:
 Size doesn't matter if it doesn't work.
I use it all the time, it works fine. It does lack the features that make makefiles incomprehensible, though.
I use it none of the time, and loath the fact it is named "make" which makes it's way into PATH. It's why it was replaced in DMD, it doesn't do what it needs to. You mean comprehensible DM make files like this? $G/nteh.obj : $C\rtlsym.h $C\nteh.c $(CC) -c -o$ $(MFLAGS) $C\nteh $G/os.obj : $C\os.c $(CC) -c -o$ $(MFLAGS) $C\os $G/out.obj : $C\out.c $(CC) -c -o$ $(MFLAGS) $C\out $G/outbuf.obj : $C\outbuf.h $C\outbuf.c $(CC) -c -o$ $(MFLAGS) $C\outbuf $G/pdata.obj : $C\pdata.c $(CC) -c -o$ $(MFLAGS) $C\pdata $G/ph2.obj : $C\ph2.c $(CC) -c -o$ $(MFLAGS) $C\ph2 Yes so clear.
Jun 21
prev sibling parent reply MoonlightSentinel <moonlightsentinel disroot.org> writes:
On Friday, 12 June 2020 at 00:24:41 UTC, 12345swordy wrote:
 Quite curious that I never see any attempt on it what so ever.
I assume you are talking about DMD's test runner (https://github.com/dlang/dmd/blob/master/test/run.d). There are no concrete plan's AFAICT but I generally like the idea to replace make for the other repos as well.
Jun 12
parent reply 12345swordy <alexanderheistermann gmail.com> writes:
On Friday, 12 June 2020 at 08:27:07 UTC, MoonlightSentinel wrote:
 On Friday, 12 June 2020 at 00:24:41 UTC, 12345swordy wrote:
 Quite curious that I never see any attempt on it what so ever.
I assume you are talking about DMD's test runner (https://github.com/dlang/dmd/blob/master/test/run.d). There are no concrete plan's AFAICT but I generally like the idea to replace make for the other repos as well.
I'm Sorry, I was referring to this: https://github.com/dlang/dmd/blob/master/src/build.d
Jun 12
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/12/20 9:42 AM, 12345swordy wrote:
 On Friday, 12 June 2020 at 08:27:07 UTC, MoonlightSentinel wrote:
 On Friday, 12 June 2020 at 00:24:41 UTC, 12345swordy wrote:
 Quite curious that I never see any attempt on it what so ever.
I assume you are talking about DMD's test runner (https://github.com/dlang/dmd/blob/master/test/run.d). There are no concrete plan's AFAICT but I generally like the idea to replace make for the other repos as well.
I'm Sorry, I was referring to this: https://github.com/dlang/dmd/blob/master/src/build.d
That should be killed with fire. I have seldom disliked a program this much.
Jun 12
next sibling parent reply Seb <seb wilzba.ch> writes:
On Saturday, 13 June 2020 at 03:33:14 UTC, Andrei Alexandrescu 
wrote:
 On 6/12/20 9:42 AM, 12345swordy wrote:
 On Friday, 12 June 2020 at 08:27:07 UTC, MoonlightSentinel 
 wrote:
 On Friday, 12 June 2020 at 00:24:41 UTC, 12345swordy wrote:
 Quite curious that I never see any attempt on it what so 
 ever.
I assume you are talking about DMD's test runner (https://github.com/dlang/dmd/blob/master/test/run.d). There are no concrete plan's AFAICT but I generally like the idea to replace make for the other repos as well.
I'm Sorry, I was referring to this: https://github.com/dlang/dmd/blob/master/src/build.d
That should be killed with fire. I have seldom disliked a program this much.
Huh. The Andrei I remember approved the migration away from the DigitalMars Makefiles multiple times. On the contrary, there are plans to expand build.d for druntime+phobos and the only reason it didn't happen is because I myself consider Phobos as low priority to maintain as it's basically frozen/dead code. Anyhow, PRs for build.d at druntime/phobos would be merged if someone is interested in this.
Jun 13
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/13/20 4:30 AM, Seb wrote:
 I myself consider Phobos as low priority to maintain as it's basically 
 frozen/dead code.
I was wondering what would be the drawbacks of defining an ultra-simple convention for versions of the standard library - with yearly granularity. Not being an expert in versioning I've always been coy to mention it, but how about trying it instead of the current stalemate. After all C++ does it (both with 3-year language versions and with things like std::tr1, std::tr2 etc) and it has a more dangerous modularity mechanism. D's modularity is, or should be, rock solid. So then we could simply define "vYEAR" as versions of standard library modules. So: // import the backward-compatible lib import std.algorithm; // import this year's algorithm with breaking changes import std.v2020.algorithm; // live dangerously, import work-in-progress import std.v2020.algorithm; Of course no need to define yearly versions for every year and every package/module. Just those that are worth adding backwards-breaking improvements. A module should not import both std.xyz and std.v2020.xyz. Or the behavior is dependent on xyz and defined clearly. I wonder how well this would work.
Jun 13
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/13/20 3:42 PM, Andrei Alexandrescu wrote:
 On 6/13/20 4:30 AM, Seb wrote:
 I myself consider Phobos as low priority to maintain as it's basically 
 frozen/dead code.
I was wondering what would be the drawbacks of defining an ultra-simple convention for versions of the standard library - with yearly granularity. Not being an expert in versioning I've always been coy to mention it, but how about trying it instead of the current stalemate. After all C++ does it (both with 3-year language versions and with things like std::tr1, std::tr2 etc) and it has a more dangerous modularity mechanism. D's modularity is, or should be, rock solid. So then we could simply define "vYEAR" as versions of standard library modules. So: // import the backward-compatible lib import std.algorithm; // import this year's algorithm with breaking changes import std.v2020.algorithm; // live dangerously, import work-in-progress import std.v2020.algorithm;
Eh, meant v2021 here.
Jun 13
parent reply jmh530 <john.michael.hall gmail.com> writes:
On Saturday, 13 June 2020 at 19:52:16 UTC, Andrei Alexandrescu 
wrote:
 [snip]
 D's modularity is, or should be, rock solid. So then we could 
 simply define "vYEAR" as versions of standard library modules. 
 So:
 
 // import the backward-compatible lib
 import std.algorithm;
 // import this year's algorithm with breaking changes
 import std.v2020.algorithm;
 // live dangerously, import work-in-progress
 import std.v2020.algorithm;
Eh, meant v2021 here.
Something like below allows the user to specify what they want for all of phobos at the command or to import the specific one they want. module std.algorithm; version(std_latest) { version = std_2_92; } version(std_stable) { version = std_2_89; } version(std_2019) { version = std_2_89; } version(std_2_92) { public import 2_92.std.algorithm; } version(std_2_91) { public import 2_91.std.algorithm; } version(std_2_90) { public import 2_90.std.algorithm; } version(std_2_89) { public import 2_89.std.algorithm; } ... One problem is that if you allow people to make bug fixes to old phobos versions, then there is no way for the subsequent releases to get the updates as if you could with a git branch. You would have to do them all manually. I also saw a suggestion about C++'s epoch proposal on another thread that was interesting. http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p1881r0.html
Jun 13
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/13/20 4:37 PM, jmh530 wrote:
 One problem is that if you allow people to make bug fixes to old phobos 
 versions, then there is no way for the subsequent releases to get the 
 updates as if you could with a git branch. You would have to do them all 
 manually.
It's not that bad. Typical newer iterations of phobos versions would simply list as public the unchanged symbols, so they are "deltas". Fixes to unchanged symbols would just propagate. Indeed fixes to changed symbols would need to be propagated manually, but even that may be mitigated by having the newer versions call the older versions on occasion. More importantly versioning allows us to experiment with things such as "let's use assert and Expected instead of throwing throughout". These are definitely breaking APIs, but can be implemented easily in such a way that both the backward-compatible version and the new version share the same codebase. (You can wrap noexcept code in code that throws, etc.)
 I also saw a suggestion about C++'s epoch proposal on another thread 
 that was interesting.
 http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p1881r0.html
Thanks!
Jun 13
prev sibling parent reply Jesse Phillips <Jesse.K.Phillips+D gmail.com> writes:
On Saturday, 13 June 2020 at 20:37:59 UTC, jmh530 wrote:
 One problem is that if you allow people to make bug fixes to 
 old phobos versions, then there is no way for the subsequent 
 releases to get the updates as if you could with a git branch. 
 You would have to do them all manually.
Git will allow you to create a patch from one file and apply it to another. Still manual. Working in std.experimental I found the private symbols had to work with. That is likely not as bad with this since the idea would be to recreate the phobos ecosystem around the changes. To that end it may be best to take a module and it's entirety into the new version and forwarding only happens across modules.
Jun 18
parent reply jmh530 <john.michael.hall gmail.com> writes:
On Thursday, 18 June 2020 at 14:16:13 UTC, Jesse Phillips wrote:
 On Saturday, 13 June 2020 at 20:37:59 UTC, jmh530 wrote:
 One problem is that if you allow people to make bug fixes to 
 old phobos versions, then there is no way for the subsequent 
 releases to get the updates as if you could with a git branch. 
 You would have to do them all manually.
Git will allow you to create a patch from one file and apply it to another. Still manual. [snip]
I think Andrei had made a good point about propagating up changes. This would imply keeping the structure that I had previously for std.algorithm overall, but modify the N_NN module files. For instance, if one were starting from 2_89, then that would be unchanged, while 2_90 would be module 2_90.std.algorithm; public import 2_89.std.algorithm: funA, funC, funD...; void funB() {} and so on up. For an even more fine-grained basis, you could do module 2_90_0.std.algorithm; public import 2_89.std.algorithm: funA, funC, funD...; void funB() {} module 2_90_1.std.algorithm; public import 2_90_0.std.algorithm: funA, funC, funD...; void funB() {} module 2_90.std.algorithm; public import 2_90_1.std.algorithm; So this would work pretty well in terms of minimizing the amount of code re-use going on. (there is probably scope for a mixin that can do those imports, or a new language feature "import module: -f" that imports everything except f) With respect to your suggestion, suppose you release some patch for funB in 2.90.1 late, i.e. when the current version is 2.95 or something. If funB has not been changed over this period, then it propagates up without issue (note that the import of 2_89 would not be over the patch, 2_89 would be structured like 2_90). However, if funB has changed, then future updates will not benefit from it. You would have to manually update those as well. Your suggestion would be useful in this case, I think.
Jun 18
next sibling parent reply Jesse Phillips <Jesse.K.Phillips+D gmail.com> writes:
On Thursday, 18 June 2020 at 15:55:06 UTC, jmh530 wrote:
 not benefit from it. You would have to manually update those as 
 well. Your suggestion would be useful in this case, I think.
I think my concern is mostly moot. I think managing the updates will be a challenge in all approaches. But I also think this is a good idea overall.
Jun 18
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/18/20 10:08 PM, Jesse Phillips wrote:
 On Thursday, 18 June 2020 at 15:55:06 UTC, jmh530 wrote:
 not benefit from it. You would have to manually update those as well. 
 Your suggestion would be useful in this case, I think.
I think my concern is mostly moot. I think managing the updates will be a challenge in all approaches. But I also think this is a good idea overall.
One good goal for std.v2020 would be to forego autodecoding throughout. The endeavor will no doubt present challenges, which in turn will prompt clever solutions and a good accumulation of experience.
Jun 19
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:
[...]
 One good goal for std.v2020 would be to forego autodecoding
 throughout.
[...] Another could be to fix up the range API -- i.e, reconsider the ugliness that is .save, now that D has copy ctors. T -- I'm still trying to find a pun for "punishment"...
Jun 19
next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:
 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu 
 via Digitalmars-d wrote: [...]
 One good goal for std.v2020 would be to forego autodecoding 
 throughout.
[...] Another could be to fix up the range API -- i.e, reconsider the ugliness that is .save, now that D has copy ctors.
How do you see that? Fundamentally, what have copy ctors changed in this regard?
Jun 20
next sibling parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Saturday, 20 June 2020 at 09:49:38 UTC, Stanislav Blinov wrote:
 On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:
 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu 
 via Digitalmars-d wrote: [...]
 One good goal for std.v2020 would be to forego autodecoding 
 throughout.
[...] Another could be to fix up the range API -- i.e, reconsider the ugliness that is .save, now that D has copy ctors.
How do you see that? Fundamentally, what have copy ctors changed in this regard?
I suppose that postblit has had various implementation and language design issues (which copy constructors address), which prevented them from being a reliable alternative to save(). This and probably also class support.
Jun 20
next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 20 June 2020 at 10:03:06 UTC, Petar Kirov 
[ZombineDev] wrote:
 On Saturday, 20 June 2020 at 09:49:38 UTC, Stanislav Blinov 
 wrote:
 On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:
 Another could be to fix up the range API -- i.e, reconsider 
 the ugliness that is .save, now that D has copy ctors.
How do you see that? Fundamentally, what have copy ctors changed in this regard?
I suppose that postblit has had various implementation and language design issues (which copy constructors address), which prevented them from being a reliable alternative to save(). This and probably also class support.
All ranges are supposed to be copyable, so their copy-ability is not sufficient to distinguish input and forward ranges. Whether it's postblit or copy ctor makes no difference here, unless I'm missing something.
Jun 20
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Saturday, 20 June 2020 at 10:52:37 UTC, Stanislav Blinov wrote:
 On Saturday, 20 June 2020 at 10:03:06 UTC, Petar Kirov 
 [ZombineDev] wrote:
 On Saturday, 20 June 2020 at 09:49:38 UTC, Stanislav Blinov 
 wrote:
 On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:
 Another could be to fix up the range API -- i.e, reconsider 
 the ugliness that is .save, now that D has copy ctors.
How do you see that? Fundamentally, what have copy ctors changed in this regard?
I suppose that postblit has had various implementation and language design issues (which copy constructors address), which prevented them from being a reliable alternative to save(). This and probably also class support.
All ranges are supposed to be copyable, so their copy-ability is not sufficient to distinguish input and forward ranges. Whether it's postblit or copy ctor makes no difference here, unless I'm missing something.
As far as I know, back when the ranges API was worked on postblit ctors didn't work reliably, so .save() was the only option. If it had worked, we could require that non-forward ranges are non-copyable.
Jun 20
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 20 June 2020 at 11:20:13 UTC, Petar Kirov 
[ZombineDev] wrote:

 As far as I know, back when the ranges API was worked on 
 postblit ctors didn't work reliably, so .save() was the only 
 option.
Funny that, at the moment it's the copy ctors that aren't working reliably (e.g. they're not working at all with arrays). :)
 If it had worked, we could require that non-forward ranges are 
 non-copyable.
I can imagine this would be quite some work to adapt all of Phobos to *that*. I mean, things like auto rem = makeSomeInputRange.find!pred; auto flt = makeSomeInputRange.filter!pred; Those would not compile. They could be made to compile by `move`ing the argument for the return. But then you still won't be able to pass those results around, unless via a refRange or `move`.
Jun 20
next sibling parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Saturday, 20 June 2020 at 12:30:43 UTC, Stanislav Blinov wrote:
 On Saturday, 20 June 2020 at 11:20:13 UTC, Petar Kirov 
 [ZombineDev] wrote:

 As far as I know, back when the ranges API was worked on 
 postblit ctors didn't work reliably, so .save() was the only 
 option.
Funny that, at the moment it's the copy ctors that aren't working reliably (e.g. they're not working at all with arrays). :)
You mean that array.dup doesn't work if the element type uses copy constructors? I haven't checked so I guess it's a problem indeed, though this shouldn't affect the ranges themselves (i.e. array slices).
 If it had worked, we could require that non-forward ranges are 
 non-copyable.
I can imagine this would be quite some work to adapt all of Phobos to *that*. I mean, things like auto rem = makeSomeInputRange.find!pred; auto flt = makeSomeInputRange.filter!pred; Those would not compile. They could be made to compile by `move`ing the argument for the return. But then you still won't be able to pass those results around, unless via a refRange or `move`.
Yeah, I know. Though input-only range are odd in general. I think they can only work well either if the code moves them around, or if they use reference semantics (classes or refRange).
Jun 20
next sibling parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 20 June 2020 at 13:01:33 UTC, Petar Kirov 
[ZombineDev] wrote:

 You mean that array.dup doesn't work if the element type uses 
 copy constructors? I haven't checked so I guess it's a problem 
 indeed, though this shouldn't affect the ranges themselves 
 (i.e. array slices).
Yes, Razvan Nitu is working on a fix for .dup already. But there's lots more. The ~= also doesn't call copy ctor. Nor does copying of slices and static arrays, i.e. S[10] arr; auto brr = arr; // no copy ctors called arr[0 .. 2] = arr[2 .. 4]; // no copy ctors called While .dup/.idup are a druntime-only fix, the rest requires fixes in the compiler. It may be of small concern to ranges themselves (barring ranges-of-ranges), but a concern nonetheless.
 Yeah, I know. Though input-only range are odd in general. I 
 think they can only work well either if the code moves them 
 around, or if they use reference semantics (classes or 
 refRange).
But if they *do* use reference semantics, they need be copyable (see e.g. std.stdio.ByLineImpl). Requiring them to become classes I'd say would be too harsh. That said, I agree that most of them would be a consume-and-throw-away (i.e. temporaries). But there's always an odd case where you'd want to consume them in staggered steps.
Jun 20
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/20 9:01 AM, Petar Kirov [ZombineDev] wrote:
 On Saturday, 20 June 2020 at 12:30:43 UTC, Stanislav Blinov wrote:
 On Saturday, 20 June 2020 at 11:20:13 UTC, Petar Kirov [ZombineDev] 
 wrote:

 As far as I know, back when the ranges API was worked on postblit 
 ctors didn't work reliably, so .save() was the only option.
Funny that, at the moment it's the copy ctors that aren't working reliably (e.g. they're not working at all with arrays). :)
You mean that array.dup doesn't work if the element type uses copy constructors? I haven't checked so I guess it's a problem indeed, though this shouldn't affect the ranges themselves (i.e. array slices).
I thought this was fixed a while ago.
Jun 20
parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 20 June 2020 at 14:54:49 UTC, Andrei Alexandrescu 
wrote:
 On 6/20/20 9:01 AM, Petar Kirov [ZombineDev] wrote:
 You mean that array.dup doesn't work if the element type uses 
 copy constructors? I haven't checked so I guess it's a problem 
 indeed, though this shouldn't affect the ranges themselves 
 (i.e. array slices).
I thought this was fixed a while ago.
Huh? https://github.com/dlang/druntime/pull/3139#discussion_r441924024 :)
Jun 20
prev sibling next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Saturday, 20 June 2020 at 12:30:43 UTC, Stanislav Blinov wrote:
 If it had worked, we could require that non-forward ranges are 
 non-copyable.
I can imagine this would be quite some work to adapt all of Phobos to *that*. I mean, things like auto rem = makeSomeInputRange.find!pred; auto flt = makeSomeInputRange.filter!pred; Those would not compile. They could be made to compile by `move`ing the argument for the return. But then you still won't be able to pass those results around, unless via a refRange or `move`.
https://github.com/WalterBright/DIPs/blob/13NNN-WGB.md/DIPs/13NNN-WGB.md
Jun 20
parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 20 June 2020 at 13:09:54 UTC, Paul Backus wrote:

 https://github.com/WalterBright/DIPs/blob/13NNN-WGB.md/DIPs/13NNN-WGB.md
I'm well aware, that one is a much needed enhancement. But it's a long ways away, and would have to be adapted to through quite a process of its own.
Jun 20
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/20/20 8:30 AM, Stanislav Blinov wrote:
 On Saturday, 20 June 2020 at 11:20:13 UTC, Petar Kirov [ZombineDev] wrote:
 
 As far as I know, back when the ranges API was worked on postblit 
 ctors didn't work reliably, so .save() was the only option.
Funny that, at the moment it's the copy ctors that aren't working reliably (e.g. they're not working at all with arrays). :)
 If it had worked, we could require that non-forward ranges are 
 non-copyable.
I can imagine this would be quite some work to adapt all of Phobos to *that*. I mean, things like auto rem = makeSomeInputRange.find!pred; auto flt = makeSomeInputRange.filter!pred; Those would not compile. They could be made to compile by `move`ing the argument for the return. But then you still won't be able to pass those results around, unless via a refRange or `move`.
You shouldn't need move for rvalues, which those should be. Algorithms that support non-forward ranges should be able to cope with using move where required on their parameters. There have been a few libraries released that use non-copyable structs to model things that making copies of will prove problematic. For instance, file handles. And people seem to be able to use these reasonably well. But the larger point is that true input-only ranges are rare. The only problem is for classes, since you cannot invoke a copy constructor on those. It would be a drastic change, I don't see it happening. But the status quo of save() is pretty bad as well. -Steve
Jun 20
next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 20 June 2020 at 18:26:43 UTC, Steven Schveighoffer 
wrote:
 On 6/20/20 8:30 AM, Stanislav Blinov wrote:
 I can imagine this would be quite some work to adapt all of 
 Phobos to *that*. I mean, things like
 
 auto rem = makeSomeInputRange.find!pred;
 auto flt = makeSomeInputRange.filter!pred;
 
 Those would not compile. They could be made to compile by 
 `move`ing the argument for the return. But then you still 
 won't be able to pass those results around, unless via a 
 refRange or `move`.
You shouldn't need move for rvalues, which those should be. Algorithms that support non-forward ranges should be able to cope with using move where required on their parameters.
`find` returns the remainder of the range, i.e it's (pseudocode): auto find(alias pred,R)(R r) { while (!r.empty && !pred(r.front)) r.popFront; return r; } There should be a `move` at that return. There's no NRVO, the compiler makes a copy there. Which, of course, it wouldn't be able to if R was non-copyable :) It *may* become a move under Walter's proposal: auto find(alias pred,R)(return R r) { while (!r.empty && !pred(r.front)) r.popFront; return r; // return what's called a `move ref` in the proposal } but until then?..
 There have been a few libraries released that use non-copyable 
 structs to model things that making copies of will prove 
 problematic. For instance, file handles. And people seem to be 
 able to use these reasonably well.
We're talking ranges here, not file handles :) Most algorithms in Phobos take ranges by value.
 But the larger point is that true input-only ranges are rare. 
 The only problem is for classes, since you cannot invoke a copy 
 constructor on those.

 It would be a drastic change, I don't see it happening. But the 
 status quo of save() is pretty bad as well.
*Why* is it bad?
Jun 20
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/20/20 3:16 PM, Stanislav Blinov wrote:
 On Saturday, 20 June 2020 at 18:26:43 UTC, Steven Schveighoffer wrote:
 On 6/20/20 8:30 AM, Stanislav Blinov wrote:
 I can imagine this would be quite some work to adapt all of Phobos to 
 *that*. I mean, things like

 auto rem = makeSomeInputRange.find!pred;
 auto flt = makeSomeInputRange.filter!pred;

 Those would not compile. They could be made to compile by `move`ing 
 the argument for the return. But then you still won't be able to pass 
 those results around, unless via a refRange or `move`.
You shouldn't need move for rvalues, which those should be. Algorithms that support non-forward ranges should be able to cope with using move where required on their parameters.
`find` returns the remainder of the range, i.e it's (pseudocode): auto find(alias pred,R)(R r) {     while (!r.empty && !pred(r.front))         r.popFront;     return r; } There should be a `move` at that return. There's no NRVO, the compiler makes a copy there.  Which, of course, it wouldn't be able to if R was non-copyable :)
Yeah, it needs to use move. But the usage doesn't change, which is what I thought you were referring to.
 It *may* become a move under Walter's proposal:
 
 auto find(alias pred,R)(return R r)
 {
      while (!r.empty && !pred(r.front))
          r.popFront;
      return r; // return what's called a `move ref` in the proposal
 }
 
 but until then?..
You'd have to modify Phobos.
 
 There have been a few libraries released that use non-copyable structs 
 to model things that making copies of will prove problematic. For 
 instance, file handles. And people seem to be able to use these 
 reasonably well.
We're talking ranges here, not file handles :) Most algorithms in Phobos take ranges by value.
Which means that you have to destroy the original, because it's no longer valid. Today, you do: auto otherRange = inputRange.find(something); use(inputRange); // invalid, but compiler lets you. And it probably messes up otherRange too.
 
 But the larger point is that true input-only ranges are rare. The only 
 problem is for classes, since you cannot invoke a copy constructor on 
 those.

 It would be a drastic change, I don't see it happening. But the status 
 quo of save() is pretty bad as well.
*Why* is it bad?
Because there's no enforcement. You can copy most forward ranges without using save, and it works EXACTLY the same. In fact, most save implementations are {return this;}. If you ever use a forward range that requires the use of .save to actually copy it, you will have problems. So it essentially promotes generic code that is incorrect, but works fine for most cases. I'm positive there are still cases like this in Phobos. It's very easy and effortless to make a copy without using .save. When's the last time you saw: foreach(x; someLvalueRange.save) Because if you don't see that, for a forward range, it's an incorrect usage. -Steve
Jun 20
parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 20 June 2020 at 20:09:56 UTC, Steven Schveighoffer 
wrote:
 On 6/20/20 3:16 PM, Stanislav Blinov wrote:
 There should be a `move` at that return. There's no NRVO, the 
 compiler makes a copy there.  Which, of course, it wouldn't be 
 able to if R was non-copyable :)
Yeah, it needs to use move. But the usage doesn't change, which is what I thought you were referring to.
The usage does change. Because then (i.e. if input ranges were to be made non-copyable): auto r = makeInputRange; auto rem = r.find!pred; ...no longer compiles. And *that* will be all over Phobos, which is my original point about required massive changes. Which, as it turns out, may not be that bad of a thing in principle (though, of course, going over all parts of Phobos that touch ranges to fix them would be a huge endeavor).
 It *may* become a move under Walter's proposal:
 but until then?..
You'd have to modify Phobos.
Exactly.
 There have been a few libraries released that use 
 non-copyable structs to model things that making copies of 
 will prove problematic. For instance, file handles. And 
 people seem to be able to use these reasonably well.
We're talking ranges here, not file handles :) Most algorithms in Phobos take ranges by value.
 Which means that you have to destroy the original, because it's 
 no longer valid.

 Today, you do:

 auto otherRange = inputRange.find(something);

 use(inputRange); // invalid, but compiler lets you. And it 
 probably messes up otherRange too.
Yeah, that's a good point, thanks.
 But the larger point is that true input-only ranges are rare. 
 The only problem is for classes, since you cannot invoke a 
 copy constructor on those.

 It would be a drastic change, I don't see it happening. But 
 the status quo of save() is pretty bad as well.
*Why* is it bad?
Because there's no enforcement. You can copy most forward ranges without using save, and it works EXACTLY the same. In fact, most save implementations are {return this;}. If you ever use a forward range that requires the use of .save to actually copy it, you will have problems. So it essentially promotes generic code that is incorrect, but works fine for most cases. I'm positive there are still cases like this in Phobos. It's very easy and effortless to make a copy without using .save. When's the last time you saw: foreach(x; someLvalueRange.save) Because if you don't see that, for a forward range, it's an incorrect usage.
In that case, making input ranges non-copyable *may* indeed be a solution.
Jun 20
prev sibling next sibling parent reply Jon Degenhardt <jond noreply.com> writes:
On Saturday, 20 June 2020 at 18:26:43 UTC, Steven Schveighoffer 
wrote:
 But the larger point is that true input-only ranges are rare. 
 The only problem is for classes, since you cannot invoke a copy 
 constructor on those.
Interesting discussion. Could you expand on this comment? Several people have mentioned this. I write my own input ranges somewhat regularly. I've never had the need to make them forward ranges. However, the typical reason for creating a range is because I have application specific data that I want to iterate over (and usually construct) lazily. Input ranges are very convenient way to do this. I do end up making many of them reference ranges though. So, I'm wondering if its really that input-only ranges are rare, or if it's that the number of algorithms that can be used on input-only ranges is small. Or perhaps I'm not quite grokking the distinction between a "true" input-only range and one that satisfies isInputRange, but none of the other range primitive tests. --Jon
Jun 20
next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Saturday, 20 June 2020 at 19:23:55 UTC, Jon Degenhardt wrote:
 On Saturday, 20 June 2020 at 18:26:43 UTC, Steven Schveighoffer 
 wrote:
 But the larger point is that true input-only ranges are rare. 
 The only problem is for classes, since you cannot invoke a 
 copy constructor on those.
Interesting discussion. Could you expand on this comment? Several people have mentioned this. I write my own input ranges somewhat regularly. I've never had the need to make them forward ranges. However, the typical reason for creating a range is because I have application specific data that I want to iterate over (and usually construct) lazily. Input ranges are very convenient way to do this. I do end up making many of them reference ranges though. So, I'm wondering if its really that input-only ranges are rare, or if it's that the number of algorithms that can be used on input-only ranges is small. Or perhaps I'm not quite grokking the distinction between a "true" input-only range and one that satisfies isInputRange, but none of the other range primitive tests. --Jon
Could you easily make your ranges into forward ranges if you wanted to? That is, if you copy one of them, and then iterate the original, does the copy remain valid? If so, they're not "true" input ranges. A "true" input range is something like a file handle, where you couldn't implement `save` even if you wanted to--iterating one copy automatically invalidates all the others.
Jun 20
parent Jon Degenhardt <jond noreply.com> writes:
On Saturday, 20 June 2020 at 19:35:07 UTC, Paul Backus wrote:
 On Saturday, 20 June 2020 at 19:23:55 UTC, Jon Degenhardt wrote:
 On Saturday, 20 June 2020 at 18:26:43 UTC, Steven 
 Schveighoffer wrote:
 But the larger point is that true input-only ranges are rare. 
 The only problem is for classes, since you cannot invoke a 
 copy constructor on those.
Interesting discussion. Could you expand on this comment? Several people have mentioned this. I write my own input ranges somewhat regularly. I've never had the need to make them forward ranges. However, the typical reason for creating a range is because I have application specific data that I want to iterate over (and usually construct) lazily. Input ranges are very convenient way to do this. I do end up making many of them reference ranges though. So, I'm wondering if its really that input-only ranges are rare, or if it's that the number of algorithms that can be used on input-only ranges is small. Or perhaps I'm not quite grokking the distinction between a "true" input-only range and one that satisfies isInputRange, but none of the other range primitive tests. --Jon
Could you easily make your ranges into forward ranges if you wanted to? That is, if you copy one of them, and then iterate the original, does the copy remain valid? If so, they're not "true" input ranges. A "true" input range is something like a file handle, where you couldn't implement `save` even if you wanted to--iterating one copy automatically invalidates all the others.
Ah, thanks for this clarification. I'd have to go back and look at all the different ranges I've written, but I'd say there are a fair number in both camps. Now I'm wondering if there's a correlation with the use of reference ranges. That is, have I used reference ranges when the range falls into the "true" input range set. The answer may be yes.
Jun 20
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/20/20 3:23 PM, Jon Degenhardt wrote:
 On Saturday, 20 June 2020 at 18:26:43 UTC, Steven Schveighoffer wrote:
 But the larger point is that true input-only ranges are rare. The only 
 problem is for classes, since you cannot invoke a copy constructor on 
 those.
Interesting discussion. Could you expand on this comment? Several people have mentioned this.
There are a few "base" ranges, like arrays, and data structure ranges. The one common true "input-only" range is a stream-based range (like File.byLine). Other than that, wrapping ranges should implement the primitives that their dependencies do. I'd say arrays are the most common range, and used for almost everything. Even with file streams, most of the time, you don't use the stream as a range, but use algorithms on the buffered data (which is an array). An input range is generally useful in foreach, and almost nothing else.
 
 I write my own input ranges somewhat regularly. I've never had the need 
 to make them forward ranges. However, the typical reason for creating a 
 range is because I have application specific data that I want to iterate 
 over (and usually construct) lazily. Input ranges are very convenient 
 way to do this. I do end up making many of them reference ranges though.
Many people do not bother adding ".save" because it's an extra thing, and copying works fine even if you don't declare save.
 
 So, I'm wondering if its really that input-only ranges are rare, or if 
 it's that the number of algorithms that can be used on input-only ranges 
 is small. Or perhaps I'm not quite grokking the distinction between a 
 "true" input-only range and one that satisfies isInputRange, but none of 
 the other range primitive tests.
An input range cannot be (correctly) iterated more than once, even if you make copies of it. -Steve
Jun 20
parent reply Jon Degenhardt <jond noreply.com> writes:
On Saturday, 20 June 2020 at 20:17:29 UTC, Steven Schveighoffer 
wrote:
 On 6/20/20 3:23 PM, Jon Degenhardt wrote:
 On Saturday, 20 June 2020 at 18:26:43 UTC, Steven 
 Schveighoffer wrote:
 But the larger point is that true input-only ranges are rare. 
 The only problem is for classes, since you cannot invoke a 
 copy constructor on those.
Interesting discussion. Could you expand on this comment? Several people have mentioned this.
There are a few "base" ranges, like arrays, and data structure ranges. The one common true "input-only" range is a stream-based range (like File.byLine).
Thanks for the detailed reply, it's very helpful and I'm finding the discussion useful. Another potentially useful example is a stream of random numbers, where it may be undesirable to allow the state of the random number stream to be copied.
 An input range is generally useful in foreach, and almost 
 nothing else.
Here I might disagree a bit. Nearly all the algorithms in std.algorithm.iteration (map, filter, fold, etc.) operate on input-only ranges. These are important classes of operations. They are largely alternate forms of foreach, so likely this is more clarification than disagreement. Related thought: One-time iteration of a stream is common in many of the apps I write. I tend to think of such streams as logically input-only. Whether it's valid/legal to copy the stream at implementation level is a different question. More recently I've finding reasons to enforce at most once iteration, even if the underlying data structures (like an array) don't care. --Jon
Jun 20
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/20 5:12 PM, Jon Degenhardt wrote:
 Nearly all the algorithms in std.algorithm.iteration (map, filter, fold, 
 etc.) operate on input-only ranges.
Yah, one good experiment would be to implement these using alternate simpler APIs and see how they work. You can infer a lot by just writing code and "smell" it. I know byLine didn't smell quite right.
Jun 20
parent Jon Degenhardt <jond noreply.com> writes:
On Sunday, 21 June 2020 at 00:15:35 UTC, Andrei Alexandrescu 
wrote:
 On 6/20/20 5:12 PM, Jon Degenhardt wrote:
 Nearly all the algorithms in std.algorithm.iteration (map, 
 filter, fold, etc.) operate on input-only ranges.
Yah, one good experiment would be to implement these using alternate simpler APIs and see how they work. You can infer a lot by just writing code and "smell" it. I know byLine didn't smell quite right.
Yeah, that would be a nice focused set to try it out on.
Jun 21
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/20 3:23 PM, Jon Degenhardt wrote:
 On Saturday, 20 June 2020 at 18:26:43 UTC, Steven Schveighoffer wrote:
 But the larger point is that true input-only ranges are rare. The only 
 problem is for classes, since you cannot invoke a copy constructor on 
 those.
Interesting discussion. Could you expand on this comment? Several people have mentioned this. I write my own input ranges somewhat regularly. I've never had the need to make them forward ranges. However, the typical reason for creating a range is because I have application specific data that I want to iterate over (and usually construct) lazily. Input ranges are very convenient way to do this. I do end up making many of them reference ranges though. So, I'm wondering if its really that input-only ranges are rare, or if it's that the number of algorithms that can be used on input-only ranges is small. Or perhaps I'm not quite grokking the distinction between a "true" input-only range and one that satisfies isInputRange, but none of the other range primitive tests.
Numbers are kinda in your favor. Without getting anywhere near scientific about it: git grep isInputRange | wc -l 601 git grep isForwardRange | wc -l 369 git grep isBidirectionalRange | wc -l 167 git grep isRandomAccessRange | wc -l 285
Jun 20
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/20 2:26 PM, Steven Schveighoffer wrote:
 It would be a drastic change, I don't see it happening.
I totally see it happening if we go with versioning. D has this good module system, so it should work real nice. To the extent it doesn't, it would point to opportunities for improvement. The stalemate we're in has taken long enough.
Jun 20
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/20 6:03 AM, Petar Kirov [ZombineDev] wrote:
 On Saturday, 20 June 2020 at 09:49:38 UTC, Stanislav Blinov wrote:
 On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:
 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu via 
 Digitalmars-d wrote: [...]
 One good goal for std.v2020 would be to forego autodecoding throughout.
[...] Another could be to fix up the range API -- i.e, reconsider the ugliness that is .save, now that D has copy ctors.
How do you see that? Fundamentally, what have copy ctors changed in this regard?
I suppose that postblit has had various implementation and language design issues (which copy constructors address), which prevented them from being a reliable alternative to save(). This and probably also class support.
That's not the problem. There'd need to be an API distinction between the two that is introspectable. Many input ranges should be copyable, which makes them indistinguishable from forward range by means of copy ctor.
Jun 20
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/20 5:49 AM, Stanislav Blinov wrote:
 On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:
 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu via 
 Digitalmars-d wrote: [...]
 One good goal for std.v2020 would be to forego autodecoding throughout.
[...] Another could be to fix up the range API -- i.e, reconsider the ugliness that is .save, now that D has copy ctors.
How do you see that? Fundamentally, what have copy ctors changed in this regard?
I've been thinking of this for a while. The fact that input ranges must buffer one element (i.e. make front() callable several times) has been a gauche choice. Input ranges should have only one API: bool fetchNext(T& target); Fill the user-provided target with the next element and return true. At the end of the range, return false and leave the target alone. I'd considered this design back in the day but I was annoyed that it required a copy for ranges that in fact are better than input ranges (e.g. arrays etc). It's still a disadvantage, but over the years I realized there are not that many algorithms that only work on input ranges so we could make-do. Of those for which efficiency is a concern, they could easily be specialized for forward ranges.
Jun 20
next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 20 June 2020 at 14:51:32 UTC, Andrei Alexandrescu 
wrote:
 On 6/20/20 5:49 AM, Stanislav Blinov wrote:
 On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:
 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu 
 via Digitalmars-d wrote: [...]
 One good goal for std.v2020 would be to forego autodecoding 
 throughout.
[...] Another could be to fix up the range API -- i.e, reconsider the ugliness that is .save, now that D has copy ctors.
How do you see that? Fundamentally, what have copy ctors changed in this regard?
I've been thinking of this for a while. The fact that input ranges must buffer one element (i.e. make front() callable several times) has been a gauche choice.
I think there's a bit of conflation going on here. Being able to call front() several times does not necessitate the *range* being buffering. After all, the range, generally speaking, need not store any data at all. Buffering comes as a necessity for some *data sources*, which are either slow to fetch the data, or simply don't have it yet, not unlike the one you're showing below:
 Input ranges should have only one API:

 bool fetchNext(T& target);

 Fill the user-provided target with the next element and return 
 true. At the end of the range, return false and leave the 
 target alone.
For anything other than PODs, this API is less efficient and forces the callers to, effectively, do busy work (i.e. boilerplate just to facilitate language plumbing). while (!range.empty) { dataStructure.emplace(range.front); // * range.popFront(); } As far as the range API itself is concerned, the above line (*), under Walter's "moving" proposal, is *at most* a constructor call and a move-construct (or just the constructor call, in a good optimizing compiler with the right data structure implementation), and *no* (zero!) destructor calls. Mind you, the `front` need not, necessarily, be returning a copy. Consider e.g. a range consuming a socket: it'd just construct the element out of buffered data and return it with NRVO (IOW, construct it directly on caller's stack, or, potentially, straight in the `dataStructure`). Versus this: while (true) { ElementType tmp = ElementType.init; if (range.fetchNext(tmp)) // That's a copy-assign, i.e. a (redundant) dtor + copy dataStructure.emplace(move(tmp)); // one (or two) move-constructs else break; } // redundant dtor I am by no means an expert on compilers, but something tells me that eliding redundant calls and copies from *that* isn't a task for the light-hearted. I'll take a "buffering" API over that any day. Not that the `fetchNext` API doesn't have its uses (it absolutely does), I just don't think it's a good candidate for ranges.
Jun 20
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/20 12:41 PM, Stanislav Blinov wrote:
 Being able to call front() several times does not necessitate the 
 *range* being buffering.
If one calls front() twice without advancing the range, where does the range return the value from? Input ranges must buffer one element at least (input iterators in STL, too). This has been a source of annoyance in implementing them.
Jun 20
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Sunday, 21 June 2020 at 00:05:01 UTC, Andrei Alexandrescu 
wrote:
 On 6/20/20 12:41 PM, Stanislav Blinov wrote:
 Being able to call front() several times does not necessitate 
 the *range* being buffering.
If one calls front() twice without advancing the range, where does the range return the value from?
The most primitive example? The range is lazy and builds the element on every call. But I believe I've tried to make the distinction clear: *range* itself isn't necessarily buffering. Whatever it's iterating over may have to. And, in case that wasn't clear either, I understand perfectly where you're coming from, since typical input ranges do indeed have to memoize that nasty first element upon initialization (and since they have to do that, they have to repeat that for every element thenceforth). Thing is, the input stream `bool fetchNext(ref T);` does not solve this problem, it merely shrugs it off unto the caller, with precarious consequences, as demonstrated.
Jun 20
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/20 8:38 PM, Stanislav Blinov wrote:
 On Sunday, 21 June 2020 at 00:05:01 UTC, Andrei Alexandrescu wrote:
 On 6/20/20 12:41 PM, Stanislav Blinov wrote:
 Being able to call front() several times does not necessitate the 
 *range* being buffering.
If one calls front() twice without advancing the range, where does the range return the value from?
The most primitive example? The range is lazy and builds the element on every call. But I believe I've tried to make the distinction clear: *range* itself isn't necessarily buffering. Whatever it's iterating over may have to. And, in case that wasn't clear either, I understand perfectly where you're coming from, since typical input ranges do indeed have to memoize that nasty first element upon initialization (and since they have to do that, they have to repeat that for every element thenceforth). Thing is, the input stream `bool fetchNext(ref T);` does not solve this problem, it merely shrugs it off unto the caller, with precarious consequences, as demonstrated.
I appreciate there's no shortage of people who teach my design and my code back to me. I honestly do, a lot. It's unclear that much of anything, was demonstrated. Was the better alternative to make input ranges noncopyable? It only takes a couple of hours of coding a bit of that up to see it's quite onerous.
Jun 21
next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Sunday, 21 June 2020 at 14:47:55 UTC, Andrei Alexandrescu 
wrote:

 I appreciate there's no shortage of people who teach my design 
 and my code back to me. I honestly do, a lot.
And I, in turn, appreciate that you nicked one paragraph and ignored (I assume dismissed) everything else. Very motivating.
 It's unclear that much of anything, was demonstrated. Was the 
 better alternative to make input ranges noncopyable?
No. No. No. It wasn't an alternative to your API. We weren't even discussing a completely alternative API until you threw it in. I compared of two APIs in terms of operations. But "nothing was demonstrated". OK! Anyway, the whole discussion was going around the existing API, and either: - input ranges non-copyable and forward ranges are, or - status quo (i.e. save() for forward ranges) I didn't catch on the distinction initially, when H.S. brought up the save(); now I do. TLDR for all of the below - under either API, input ranges being non-copyable is a much better choice. Input ranges, by nature being one-pass, *should not be copyable*. You can't do anything (good) with a copy, and have to invest into implementing a copy that won't bite. If you're giving such range away - you're giving it *away*, to someone else to consume. It being copyable only means that you're leaving for yourself a mutable reference to state that you shouldn't touch again. When you need the remainder back - your callee will move it back. You mentioned the "smell" of ByLine. A good deal of that "smell" emanates from its copy-ability. I mean, disabling a constructor versus reference-counted internals - which plate is heavier?... Using either API, that smell will stay if it is to remain copyable. Most of the time you'd be using input ranges as rvalues. In the remaining cases the compiler will give you an error if you try to copy - and that's a good error.
 It only takes a couple of hours of coding a bit of that up to 
 see it's quite onerous.
It isn't. Bugs in the language and Phobos aside - it's one `move` away. Here, a few Phobos algorithms implemented with the `fetchNext` API: https://gist.github.com/radcapricorn/d76d29c6df6fa822d7889e799937f39d The good? An actual source range (ByLine) is dumb-as-a-cork-simple. No buffering, no ref-counted internals: read, slice, rinse and repeat. The bad? Well, you can see for yourself.
Jun 21
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 6/21/20 1:43 PM, Stanislav Blinov wrote:
 Input ranges, by nature being one-pass, *should not be copyable*. You 
 can't do anything (good) with a copy, and have to invest into 
 implementing a copy that won't bite. If you're giving such range away - 
 you're giving it *away*, to someone else to consume. It being copyable 
 only means that you're leaving for yourself a mutable reference to state 
 that you shouldn't touch again. When you need the remainder back - your 
 callee will move it back.
Good arguments, no doubt, but a long experience with noncopyable C++ objects suggests that defining such types need to be approached with trepidation as they are very cumbersome to use. (Any type containing a noncopyable type as a member becomes itself noncopyable; this is not something that we can inflict lightly on the casual users of input ranges. (To wit: no input streams or iterators in C++ are noncopyable, although the same argument would apply to them as well.) Also, it is not unheard of to have two input ranges fed from the same source with the obvious semantics (whoever calls functions to get more data will get the data and push the cursor further). True, buffering is an issue (what if two copies have each their own buffers?) but that's an engineering problem, not one of principles. It is quite clear to me that we can't propose a design with noncopyable input ranges without effectively making them pariahs that everybody will take pains to use and do their best to avoid.
Jun 22
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Monday, 22 June 2020 at 16:03:54 UTC, Andrei Alexandrescu 
wrote:
 On 6/21/20 1:43 PM, Stanislav Blinov wrote:
 Input ranges, by nature being one-pass, *should not be 
 copyable*...
 Good arguments, no doubt, but a long experience with 
 noncopyable C++ objects suggests that defining such types need 
 to be approached with trepidation as they are very cumbersome 
 to use.
In C++ - traditionally they were cumbersome, even to define, until move came along. In D though? The biggest annoying obstacle is Phobos making lots of redundant copies, needlessly prohibiting the use of non-copyables. That, however, is not a fault of non-copyables, but of Phobos. The GC limits the use of non-copyables for sure, though that's largely irrelevant to ranges themselves, esp. considering that classes indeed shouldn't ever be ranges.
 (Any type containing a noncopyable type as a member becomes 
 itself noncopyable;
And that is *good*.
 this is not something that we can inflict lightly on the casual 
 users of input ranges.
That is exactly what we *should* inflict on "casual" users. Compile-time errors. For trying to casually write bugs.
 (To wit: no input streams or iterators in C++ are noncopyable, 
 although the same argument would apply to them as well.)
An input range being non-copyable is immensely beneficial - it would statically catch a good deal of problems. It would *alleviate* the need of added complexity in implementations. Looking back at the ByLine again: RefCounted internals just to remain copyable. I don't think appeasing "casual users" by over-engineering is a good compromise. Non-copyables in D are easy to write, easy to use *correctly*, and difficult to use *incorrectly* (won't compile, after all).
 Also, it is not unheard of to have two input ranges fed from 
 the same source with the obvious semantics (whoever calls 
 functions to get more data will get the data and push the 
 cursor further). True, buffering is an issue (what if two 
 copies have each their own buffers?) but that's an engineering 
 problem, not one of principles.
That would be different instances of input ranges over the same source, not copies of the same range carrying unnecessary state and complexity (over-engineered solutions to the buffering issue). We're never consuming the same range *simultaneously* in different places. With an uncopyable, we'd just have to give it to a consumer and then have them give the remainder back to us.
 It is quite clear to me that we can't propose a design with 
 noncopyable input ranges without effectively making them 
 pariahs that everybody will take pains to use and do their best 
 to avoid.
Then we should propose a design that is not painful to use: // error: `input` cannot be copied // auto data = input.filter!somehow.map!something.array; // Ok: auto data = input.move.filter!somehow.map!something.array; If we need partial consumption, i.e. preservation of the remainder, terminal primitives can give it back to us (after all, wrapping range is holding onto it): auto data = input.move.filter!somehow.take(someAmount).map!something.array(input); The implementation of such `array` would look like this: auto array(Range, Remainder)(Range range, ref Remainder remainder) if (isInputRange!Range && is(typeof(range.source) == Remainder)) { import std.array : appender; auto builder = appender!(ElementType!Range[]); // this is assuming current API, substitute with whatever replacement API while (!range.empty) { builder.put(range.front); range.popFront(); } move(range.source, remainder); return builder[]; } I wouldn't say this qualifies as "painful", to author or use. Authors of ranges would be writing ranges, and not the machinery required to keep them copyable. Users of ranges would be using ranges, and not the hidden machinery required to keep them copyable.
Jun 24
parent reply Paul Backus <snarwin gmail.com> writes:
On Wednesday, 24 June 2020 at 11:49:11 UTC, Stanislav Blinov 
wrote:
 On Monday, 22 June 2020 at 16:03:54 UTC, Andrei Alexandrescu 
 wrote:
 It is quite clear to me that we can't propose a design with 
 noncopyable input ranges without effectively making them 
 pariahs that everybody will take pains to use and do their 
 best to avoid.
Then we should propose a design that is not painful to use: // error: `input` cannot be copied // auto data = input.filter!somehow.map!something.array; // Ok: auto data = input.move.filter!somehow.map!something.array; If we need partial consumption, i.e. preservation of the remainder, terminal primitives can give it back to us (after all, wrapping range is holding onto it): auto data = input.move.filter!somehow.take(someAmount).map!something.array(input);
IMO if the user has to manually call `move`, you have already failed at usability. It may seem "easy" or "obvious" to you and I, but for beginners, this is going to be a huge stumbling block. You write some code that looks like it should obviously work, you get a mysterious error message that "std.algorithm.whatever!(some, args).Result is not copyable because it is annotated with disable", and the solution is that you have to add ".move" to your code? Can you imagine having to explain that to someone in the Learn forum? I don't think I could do it with a straight face--it's too absurd. The only way non-copyable input ranges can work is if the compiler is able to implicitly move them in cases like the above. In other words, we would need something like Walter's "Copying, Moving, and Forwarding" DIP [1]. [1] https://github.com/WalterBright/DIPs/blob/13NNN-WGB.md/DIPs/13NNN-WGB.md
Jun 24
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Wednesday, 24 June 2020 at 13:14:22 UTC, Paul Backus wrote:
 On Wednesday, 24 June 2020 at 11:49:11 UTC, Stanislav Blinov 
 wrote:
 On Monday, 22 June 2020 at 16:03:54 UTC, Andrei Alexandrescu 
 wrote:
 It is quite clear to me that we can't propose a design with 
 noncopyable input ranges without effectively making them 
 pariahs that everybody will take pains to use and do their 
 best to avoid.
Then we should propose a design that is not painful to use: // error: `input` cannot be copied // auto data = input.filter!somehow.map!something.array; // Ok: auto data = input.move.filter!somehow.map!something.array; If we need partial consumption, i.e. preservation of the remainder, terminal primitives can give it back to us (after all, wrapping range is holding onto it): auto data = input.move.filter!somehow.take(someAmount).map!something.array(input);
IMO if the user has to manually call `move`, you have already failed at usability. It may seem "easy" or "obvious" to you and I, but for beginners, this is going to be a huge stumbling block.
Libraries shan't be for "beginners" nor "advanced users". And that use case is already a non-beginner. Beginners would write this: auto data = createSomeInput(args).filter!somehow.map!something.array; Create input, consume input, with no explicit moves in sight. Piecemeal consumption of an input range is something an "advanced" user would be doing :) I am also of the opinion that a huge reminder "YOU SHALL NOT COPY UNLESS YOU MUST" needs to be present on every page of any introductory (and not only) literature, and not just for D. The problem with lax copying extends far outside the realm of ranges. struct CopyCount { int count; this(ref typeof(this) other) { count = other.count + 1; } } CopyCount cc; writeln(cc); In my book, that should print CopyCount(0) (i.e. you print that which you can parse and get the original). That's not what Phobos writeln would print though. It won't even print CopyCount(1). People need to be taught to not squander their values, and how not to.
 You write some code that looks like it should obviously work, 
 you get a mysterious error message that 
 "std.algorithm.whatever!(some, args).Result is not copyable 
 because it is annotated with  disable"
Yup, because in that hypothetical universe you're using a type that is documented to be an input range, a category of ranges which is documented to be non-copyable, which means you just wrote what otherwise would've been a bug, but the compiler stopped you.
 and the solution is that you have to add ".move" to your code? 
 Can you imagine having to explain that to someone in the Learn 
 forum? I don't think I could do it with a straight face--it's 
 too absurd.
Nothing to it - you point such user to the documentation of ranges which states that input ranges are non-copyable, explains why they're non-copyable, and has usage examples for rvalues and lvalues. You're aware of a recent question in .learn there, where the user attempted to iterate a non-copyable range (the thread which turned into another prolonged discussion: https://forum.dlang.org/post/kpncjzadrwpvxupsdmle forum.dlang.org). A range which, by the way, was an input range. The explanations quickly went into "common pitfalls" territory. Jonathan M Davis' response was of particular interest: "In general, you should never use a range after it's been copied unless you know exactly what type of range you're dealing with and what its copying behavior is. If you want an independent copy, you need to use save." I don't think explaining that to a "beginner" is more practical than explaining that input ranges are consumed once and may be yielding values that are only valid between iterations.
 The only way non-copyable input ranges can work is if the 
 compiler is able to implicitly move them in cases like the 
 above. In other words, we would need something like Walter's 
 "Copying, Moving, and Forwarding" DIP [1].

 [1] 
 https://github.com/WalterBright/DIPs/blob/13NNN-WGB.md/DIPs/13NNN-WGB.md
The way that proposal exists at the moment, it won't help practically whenever you're not at "last use" (i.e. the second case - wanting to keep the lvalue for the remainder). Compiler will want to copy then. It'll certainly help for the first case though, as well as general implementation of ranges.
Jun 24
parent reply Paul Backus <snarwin gmail.com> writes:
On Wednesday, 24 June 2020 at 15:26:40 UTC, Stanislav Blinov 
wrote:
 Libraries shan't be for "beginners" nor "advanced users". And 
 that use case is already a non-beginner. Beginners would write 
 this:

 auto data = 
 createSomeInput(args).filter!somehow.map!something.array;

 Create input, consume input, with no explicit moves in sight. 
 Piecemeal consumption of an input range is something an 
 "advanced" user would be doing :)
Ok. Now they refactor their code: auto input = createSomeInput(args); auto data = input.filter!somehow.map!something.array; Kaboom! Compile error. This is why Walter's proposal is relevant here.
 You're aware of a recent question in .learn there, where the 
 user attempted to iterate a non-copyable range (the thread 
 which turned into another prolonged discussion: 
 https://forum.dlang.org/post/kpncjzadrwpvxupsdmle forum.dlang.org). A range
which, by the way, was an input range. The explanations quickly went into
"common pitfalls" territory. Jonathan M Davis' response was of particular
interest: "In general, you should never use a range after it's been copied
unless you know exactly what type of range you're dealing with and what its
copying behavior is. If you want an independent copy, you need to use save."
To be clear: I agree with you *in principle* that pure input ranges should be non-copyable. It's just that currently, non-copyable types are rather cumbersome to work with in D, and IMO it would be a step backwards in terms of usability to force them on users of std.algorithm and std.range. The solution is to make working with non-copyable types more ergonomic, and then make pure input ranges non-copyable. This will also improve the signal-to-noise ratio of compile errors: with automatic move-on-last-use, when the compiler complains about copying a range, you can be sure you have actually made a mistake somewhere in your logic, rather than just forgetting to type ".move".
Jun 24
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/24/20 11:55 AM, Paul Backus wrote:
 To be clear: I agree with you *in principle* that pure input ranges 
 should be non-copyable.
By what principle two input ranges should absolutely never use the same data feed?
Jun 24
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
I'm barely reading this thread, but could  live be useful with 
input ranges too?
Jun 24
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/24/20 12:25 PM, Adam D. Ruppe wrote:
 I'm barely reading this thread, but could  live be useful with input 
 ranges too?
I'm not sure, but with a no-quarter-given approach to copying input ranges, simple tasks like "feed input from this file, or from stdin, and output to this other file or to stdout" mutate from trivial into little research projects (btw by the same (misguded imho) arguments "pure" output ranges ought to also be noncopyable).
Jun 24
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Jun 24, 2020 at 12:29:13PM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:
 On 6/24/20 12:25 PM, Adam D. Ruppe wrote:
 I'm barely reading this thread, but could  live be useful with input
 ranges too?
I'm not sure, but with a no-quarter-given approach to copying input ranges, simple tasks like "feed input from this file, or from stdin, and output to this other file or to stdout" mutate from trivial into little research projects (btw by the same (misguded imho) arguments "pure" output ranges ought to also be noncopyable).
I second this. Whatever we decide to do about the input range / forward range distinction, it should absolutely not cripple input ranges or make them so different that you have to write two versions of the same algorithm just to handle both cases. For all of its flaws, the current range API does a beautiful job of unifying the two (to the extent possible, modulo well-known bugs / design limitations) so that, for the most part, you can ignore the difference. As long as .save isn't required, you could freely feed either an input range or a forward range to any range algorithm, and freely compose such algorithms, and it all Just Works(tm). I propose that whatever new design we settle on for input ranges, it should preserve this user-facing simplicity. What we want is a design where simple things are simple, and hard things are possible, not a design where simple things are hard and hard things are nigh impossible. To the extent possible, we should try to preserve symmetry between input and forward ranges. (By "symmetry" I mean that as long as the equivalent of .save isn't required, an input range and an output range ought to be transparently interchangeable. A kind of Liskov Substitution like principle applied to ranges.) T -- Latin's a dead language, as dead as can be; it killed off all the Romans, and now it's killing me! -- Schoolboy
Jun 24
prev sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Wednesday, 24 June 2020 at 16:25:31 UTC, Adam D. Ruppe wrote:
 I'm barely reading this thread, but could  live be useful with 
 input ranges too?
I don't think it can be, at least in its current form, but my reasoning behind making them non-copyable is pretty much the same as one behind live and pointers: at any given moment, there can be only one owner of an input range's iteration state. For some reason, Andrei equates that to not being able to read from the same source. ¯\_(ツ)_/¯
Jun 24
parent Paul Backus <snarwin gmail.com> writes:
On Wednesday, 24 June 2020 at 20:05:27 UTC, Stanislav Blinov 
wrote:
 On Wednesday, 24 June 2020 at 16:25:31 UTC, Adam D. Ruppe wrote:
 I'm barely reading this thread, but could  live be useful with 
 input ranges too?
I don't think it can be, at least in its current form, but my reasoning behind making them non-copyable is pretty much the same as one behind live and pointers: at any given moment, there can be only one owner of an input range's iteration state. For some reason, Andrei equates that to not being able to read from the same source. ¯\_(ツ)_/¯
I think the missing piece here is that it's entirely possible for an input range to "borrow" its iteration state from somewhere else, rather than owning it. For example, a pointer to an input range is, itself, considered an input range, because of how the `.` operator does automatic dereferencing: struct Example { int i; bool empty() { return false; } int front() { return i; } void popFront() { ++i; } } static assert(isInputRange!(Example*)); Clearly there is no need for `Example*` to be non-copyable, even if we think `Example` ought to be.
Jun 25
prev sibling next sibling parent Paul Backus <snarwin gmail.com> writes:
On Wednesday, 24 June 2020 at 16:17:58 UTC, Andrei Alexandrescu 
wrote:
 On 6/24/20 11:55 AM, Paul Backus wrote:
 To be clear: I agree with you *in principle* that pure input 
 ranges should be non-copyable.
By what principle two input ranges should absolutely never use the same data feed?
Point taken. An input range should either be non-copyable, or have reference semantics.
Jun 24
prev sibling parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Wednesday, 24 June 2020 at 16:17:58 UTC, Andrei Alexandrescu 
wrote:
 On 6/24/20 11:55 AM, Paul Backus wrote:
 To be clear: I agree with you *in principle* that pure input 
 ranges should be non-copyable.
By what principle two input ranges should absolutely never use the same data feed?
By no principle. If a given data feed can be used in such manner - then used it shall be. By two distinct input ranges that keep their own state and don't breed references to said state.
 I'm not sure, but with a no-quarter-given approach to copying 
 input ranges, simple tasks      like "feed input from this 
 file, or from stdin, and output to this other file or to 
 stdout" mutate from trivial into little research projects
I don't see how. This and everything of the same simplicity stays exactly the same: import std.stdio, std.array, std.algorithm; void main() { stdin .byLine .map!"a.length" .each!writeln; } Alleged issues materialize when you start using lvalues. Which I've already shown an example of a few posts back. Does a CsvReader really need to allocate, especially if my data is numbers? The answer in current incarnation of ranges is "yes". Well then, I'd rather write one that doesn't. Thanks but no thanks, Phobos.
 (btw by the same (misguded imho) arguments "pure" output ranges 
 ought to also be noncopyable).
Perhaps.
Jun 24
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/21/20 10:47 AM, Andrei Alexandrescu wrote:
 Was the better 
 alternative to make input ranges noncopyable? It only takes a couple of 
 hours of coding a bit of that up to see it's quite onerous.
To recap/put forth a few possible APIs for input ranges (that are not forward): bool fetchNext(ref T target); Spec: As long as the range is nonempty, assigns the next element to target and returns true. At end of range returns false without touching target. Pros: works naturally with actual input ranges such as files and sockets. Cons: does not work if T is qualified. Creates a copy of each element for forward ranges that exist in memory. That means essentially the API is impaired and most algorithms that work with input ranges would need to specialize for forward ranges, too. (FWIW this interface was first discussed before D had qualifiers). Composes so-so, consider implementations of filter (not bad) and map (not so nice). T* fetchNext(); Spec: As long as the range is nonempty, returns a pointer to the next element in the range. At end of range returns null. Pros: Simple and efficient for many ranges. Doesn't compose too well. Cons: Issues with escape analysis and safety. Sometimes the pointer is scope, sometimes it's not depending on the range. T fetchNext(ref bool done); or ref T fetchNext(ref bool done); Spec: As long as the range is nonempty, sets done to false and returns the next element in the range. At end of range sets done to true and returns an arbitrary T. Pros: works with qualified data. Cons: inefficient or needs two versions depending on ref/value. Complicates life of clients. Flag!"each" each(alias fun); Spec: calls fun(x) for each element in the range x - an efficient generalization of opApply. If fun returns No.each, stops iteration (and also returns No.each). Otherwise, returns Yes.each. See https://dlang.org/library/std/algorithm/iteration/each.html. Pros: works naturally and efficiently (assuming proper inlining) with many range types. Composes quite well, picture e.g. implementations of map and filter. Beautiful. Efficient implementation for forward ranges is immediate. Jives well with Oleg's famous take on iteration (http://okmij.org/ftp/papers/LL3-collections-talk.pdf). Cons: ranges such as map, filter etc. would need to expose both each() (for input ranges) and the existing interface (for better than input ranges). A number of algorithms would need to be redone to take advantage of the new interface. The language integration is not as nice as with the (sadly inefficient) existing opApply. At a point Sebastian Wilzbach and I discussed that several ranges should get an each() member function, but that never got finalized. Once each() is a member of prominent input range algos (can't hurt because people already can call the less efficient each() global function) and we accumulate experience with it, we can sanction it as a legit optional primitive for ranges. One more thing before I forget - we should drop classes that are ranges. They add too much complication. The functionality would still be present by wrapping a polymorphic implementation in a struct.
Jun 21
parent reply Jon Degenhardt <jond noreply.com> writes:
On Monday, 22 June 2020 at 02:52:57 UTC, Andrei Alexandrescu 
wrote:
 On 6/21/20 10:47 AM, Andrei Alexandrescu wrote:
 One more thing before I forget - we should drop classes that 
 are ranges. They add too much complication. The functionality 
 would still be present by wrapping a polymorphic implementation 
 in a struct.
I have used class-based ranges quite a bit to implement ranges with reference semantics. Classes are very convenient for this because the implementation is structurally very similar to a struct based implementation. Typically, replace 'struct' with 'final class' and modify a helper function constructing the range to use 'new'. It's amazingly easy. So - What aspects of class-based ranges is it valuable to drop? Polymorphism/inheritance? Reference semantics? Both? I haven't found a need for polymorphism in these ranges, just reference semantics. I realize that the Phobos experience is the opposite, that reference ranges are rarely used. Could be that my design choices could be modified to eliminate reference ranges. Could also be that approaches needed in general libraries are not always necessary in code targeting specific applications. Not trying to make hard claims about this. Just pointing out where I've found value. --Jon
Jun 22
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Jun 22, 2020 at 10:54:08PM +0000, Jon Degenhardt via Digitalmars-d
wrote:
 On Monday, 22 June 2020 at 02:52:57 UTC, Andrei Alexandrescu wrote:
 On 6/21/20 10:47 AM, Andrei Alexandrescu wrote:
 One more thing before I forget - we should drop classes that are
 ranges.  They add too much complication. The functionality would
 still be present by wrapping a polymorphic implementation in a
 struct.
I have used class-based ranges quite a bit to implement ranges with reference semantics. Classes are very convenient for this because the implementation is structurally very similar to a struct based implementation. Typically, replace 'struct' with 'final class' and modify a helper function constructing the range to use 'new'. It's amazingly easy.
[...] Yeah, I rely on class ranges on occasion too, when runtime polymorphism is required. They *are* cumbersome to use, though, I'll give you that. If we're going to remove class-based ranges, then whatever replaces them better have good support for runtime polymorphism, otherwise it's not gonna fly. This may be more common than people think. For example, sometimes I have a range helper: auto myFunc(R)(R r, bool cond) { if (cond) return r.map!someFunc; else return r.filter!someFilter.map!someFunc; } There is no way to make this compile (cond is only known at runtime), because the return statements return diverse types, except to wrap it in an InputRangeObject. Though I only rarely need this, it's nonetheless critical functionality because sometimes, you cannot know the exact range type until runtime, and runtime polymorphism is unavoidable. So this part of the current range API cannot be just thrown out without a viable replacement. T -- Маленькие детки - маленькие бедки.
Jun 22
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/22/20 11:12 PM, H. S. Teoh wrote:
 On Mon, Jun 22, 2020 at 10:54:08PM +0000, Jon Degenhardt via Digitalmars-d
wrote:
 On Monday, 22 June 2020 at 02:52:57 UTC, Andrei Alexandrescu wrote:
 On 6/21/20 10:47 AM, Andrei Alexandrescu wrote:
 One more thing before I forget - we should drop classes that are
 ranges.  They add too much complication. The functionality would
 still be present by wrapping a polymorphic implementation in a
 struct.
I have used class-based ranges quite a bit to implement ranges with reference semantics. Classes are very convenient for this because the implementation is structurally very similar to a struct based implementation. Typically, replace 'struct' with 'final class' and modify a helper function constructing the range to use 'new'. It's amazingly easy.
[...] Yeah, I rely on class ranges on occasion too, when runtime polymorphism is required. They *are* cumbersome to use, though, I'll give you that. If we're going to remove class-based ranges, then whatever replaces them better have good support for runtime polymorphism, otherwise it's not gonna fly. This may be more common than people think. For example, sometimes I have a range helper: auto myFunc(R)(R r, bool cond) { if (cond) return r.map!someFunc; else return r.filter!someFilter.map!someFunc; } There is no way to make this compile (cond is only known at runtime), because the return statements return diverse types, except to wrap it in an InputRangeObject.
You don't need polymorphism for that (necessarily), just function pointers/delegates. I've also done this kind of stuff with iopipes using tagged unions: https://github.com/schveiguy/httpiopipe/blob/a04d87de3aa3836c07d181263c399416ba005e7c/source/iopipe/http.d#L245-L252 Such a thing is also generalized in phobos: https://dlang.org/phobos/std_range.html#choose And can probably be extended further (not sure why the args aren't lazy). Your example rewritten: auto myFunc(R)(R r, bool cond) { return choose(cond, r.map!someFunc, r.filter!someFilter.map!someFunc); } -Steve
Jun 23
prev sibling next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, June 22, 2020 9:12:02 PM MDT H. S. Teoh via Digitalmars-d wrote:
 On Mon, Jun 22, 2020 at 10:54:08PM +0000, Jon Degenhardt via Digitalmars-d 
wrote:
 On Monday, 22 June 2020 at 02:52:57 UTC, Andrei Alexandrescu wrote:
 On 6/21/20 10:47 AM, Andrei Alexandrescu wrote:
 One more thing before I forget - we should drop classes that are
 ranges.  They add too much complication. The functionality would
 still be present by wrapping a polymorphic implementation in a
 struct.
I have used class-based ranges quite a bit to implement ranges with reference semantics. Classes are very convenient for this because the implementation is structurally very similar to a struct based implementation. Typically, replace 'struct' with 'final class' and modify a helper function constructing the range to use 'new'. It's amazingly easy.
[...] Yeah, I rely on class ranges on occasion too, when runtime polymorphism is required. They *are* cumbersome to use, though, I'll give you that. If we're going to remove class-based ranges, then whatever replaces them better have good support for runtime polymorphism, otherwise it's not gonna fly. This may be more common than people think. For example, sometimes I have a range helper: auto myFunc(R)(R r, bool cond) { if (cond) return r.map!someFunc; else return r.filter!someFilter.map!someFunc; } There is no way to make this compile (cond is only known at runtime), because the return statements return diverse types, except to wrap it in an InputRangeObject. Though I only rarely need this, it's nonetheless critical functionality because sometimes, you cannot know the exact range type until runtime, and runtime polymorphism is unavoidable. So this part of the current range API cannot be just thrown out without a viable replacement.
As far as forward ranges go, all that should be required is that the class be wrapped by a struct where the copy constructor does the equivalent of save. The class itself can then do the polymorphism like it would now. It's just that the class won't be passed around directly. But because it's wrapped in a struct, it then becomes possible to require that all forward ranges have the same copying semantics (thus allowing us to ditch save) as well as stuff like requiring that the init value be a valid, empty range (which not only is not the case right now but cannot be the case so long as we allow ranges which are classes). That being said, it is almost always a mistake to make a range a class if it can be avoided, because the allocations for save (or the copy constructor if it replaces save) incur a huge performance hit relative to what it costs to copy a range that's just a struct (e.g. the tests for dxml which use classes for the range are _very_ slow in comparison to those that use dynamic arrays or structs). So, even wrapped in a struct, it's not a great idea to use a class for a forward range, but with problems like you describe here, AFAIK, we really don't have a better solution. It's best to avoid the need for runtime polymorphism with ranges, but it's not always possible. That being said, it should work wrapped in a struct rather than having the class exposed directly. Now, as far as basic input ranges go, having them be classes isn't necessarily a problem, since basic input ranges are essentially reference types by their very nature (or at least, they can't be value types). If they could have value semantics, they could be forward ranges. In fact, in some ways, it would be nice to require that basic input ranges be classes, since then it would be easy to introspect a basic input range vs a forward range, and it would ensure that basic input ranges have full-on reference semantics, but it also would result in heap allocations that are not currently necessary - particularly in cases where it would currently work to use a pseudo-reference type rather than full-on reference type. So, while it would be very nice to be able to require that basic input ranges be classes, I doubt that it's actually reasonable to do so. Either way, it shouldn't be a problem to allow basic input ranges to be classes. It's just with forward ranges that it's a problem, but struct wrappers should solve that problem. - Jonathan M Davis
Jun 22
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/22/20 6:54 PM, Jon Degenhardt wrote:
 On Monday, 22 June 2020 at 02:52:57 UTC, Andrei Alexandrescu wrote:
 On 6/21/20 10:47 AM, Andrei Alexandrescu wrote:
 One more thing before I forget - we should drop classes that are 
 ranges. They add too much complication. The functionality would still 
 be present by wrapping a polymorphic implementation in a struct.
I have used class-based ranges quite a bit to implement ranges with reference semantics. Classes are very convenient for this because the implementation is structurally very similar to a struct based implementation. Typically, replace 'struct' with 'final class' and modify a helper function constructing the range to use 'new'. It's amazingly easy. So - What aspects of class-based ranges is it valuable to drop? Polymorphism/inheritance? Reference semantics? Both? I haven't found a need for polymorphism in these ranges, just reference semantics. I realize that the Phobos experience is the opposite, that reference ranges are rarely used. Could be that my design choices could be modified to eliminate reference ranges. Could also be that approaches needed in general libraries are not always necessary in code targeting specific applications. Not trying to make hard claims about this. Just pointing out where I've found value.
Problem is they're cumbersome to use - you need to remember to call save() in key places, and if you forget the compiler will be mum about it. If we do away with class ranges, save() can be eliminated in favor of the copy ctor. I assume a wrapper that does define reference semantics where needed would not be difficult to write.
Jun 22
next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, June 22, 2020 11:19:35 PM MDT Andrei Alexandrescu via Digitalmars-d 
wrote:
 On 6/22/20 6:54 PM, Jon Degenhardt wrote:
 On Monday, 22 June 2020 at 02:52:57 UTC, Andrei Alexandrescu wrote:
 On 6/21/20 10:47 AM, Andrei Alexandrescu wrote:
 One more thing before I forget - we should drop classes that are
 ranges. They add too much complication. The functionality would still
 be present by wrapping a polymorphic implementation in a struct.
I have used class-based ranges quite a bit to implement ranges with reference semantics. Classes are very convenient for this because the implementation is structurally very similar to a struct based implementation. Typically, replace 'struct' with 'final class' and modify a helper function constructing the range to use 'new'. It's amazingly easy. So - What aspects of class-based ranges is it valuable to drop? Polymorphism/inheritance? Reference semantics? Both? I haven't found a need for polymorphism in these ranges, just reference semantics. I realize that the Phobos experience is the opposite, that reference ranges are rarely used. Could be that my design choices could be modified to eliminate reference ranges. Could also be that approaches needed in general libraries are not always necessary in code targeting specific applications. Not trying to make hard claims about this. Just pointing out where I've found value.
Problem is they're cumbersome to use - you need to remember to call save() in key places, and if you forget the compiler will be mum about it. If we do away with class ranges, save() can be eliminated in favor of the copy ctor. I assume a wrapper that does define reference semantics where needed would not be difficult to write.
Except that if a forward range has reference semantics, then copying it is not equivalent to calling save. If we get rid of save in favor of using copy constructors, then we're requiring that all forward ranges have the same copying semantics and that that be that copying a forward range results in a copy that can be independently iterated over. Some internals could use references to be sure, but we get rid of the ability to have forward ranges which are actually reference types. They have to have the copying semantics of value types with regards to at least their iteration state. Now, one of the problems that we have with ranges as things stand is that the semantics of copying a range are unspecified and can vary wildly depending on the range's type, meaning that if a range is copied, you can't ever use it again (at least in generic code), because the state of the original is unknown after mutating the copy. It could be unchanged, be in the same state as the copy, or even be in an invalid state due to the copy mutating part of its state but not all of it. So, overall, having forward ranges require that copying do what save now does would be a considerable improvement with regards to cleaning up the copying semantics. Of course, that still leaves the problem of basic input ranges, since they can't have value semantics when copying (if they could, then they could be forward ranges). Pretty much by definition, basic input ranges are either reference types or pseudo-reference types (or the programmer was just lazy and didn't implement save). So, unfortunately, generic code that operates on both basic input ranges and forward ranges can't ever have guaranteed copying semantics. I think that we should try to nail down the copying semantics for ranges as much as we can though. Similarly, we should probably place requirements on the semantics of assigning to a range. That would almost certainly kill off RefRange, but IMHO, RefRange was a mistake anyway. - Jonathan M Davis
Jun 22
prev sibling parent Jon Degenhardt <jond noreply.com> writes:
On Tuesday, 23 June 2020 at 05:19:35 UTC, Andrei Alexandrescu 
wrote:
 On 6/22/20 6:54 PM, Jon Degenhardt wrote:
 On Monday, 22 June 2020 at 02:52:57 UTC, Andrei Alexandrescu 
 wrote:
 On 6/21/20 10:47 AM, Andrei Alexandrescu wrote:
 One more thing before I forget - we should drop classes that 
 are ranges. They add too much complication. The functionality 
 would still be present by wrapping a polymorphic 
 implementation in a struct.
I have used class-based ranges quite a bit to implement ranges with reference semantics. Classes are very convenient for this because the implementation is structurally very similar to a struct based implementation. Typically, replace 'struct' with 'final class' and modify a helper function constructing the range to use 'new'. It's amazingly easy.
Problem is they're cumbersome to use - you need to remember to call save() in key places, and if you forget the compiler will be mum about it. If we do away with class ranges, save() can be eliminated in favor of the copy ctor. I assume a wrapper that does define reference semantics where needed would not be difficult to write.
A couple thoughts. First, my uses for reference ranges have so far been on non-forward input ranges, so no 'save' function. No built-in member for copying either. I'm now wondering if there is some structural correlation between these choices and my application use cases, or if it is purely accidental. Second - If I'm designing a range intended to have reference semantics, I'd expect it to be easier to build that choice into the main implementation of the range. If the implementation needs to be in two parts, a value-based, copyable range, and a separate wrapper to define reference semantics, then I'd expect it to be more complicated. Even if the wrapper is found in the standard library. Simple things like decisions on how to organize unit tests. Are there two sets of unit tests, one for the value-based layer, and a second for wrapped layer? (I.e. If the value based layer can never be directly instantiated, should it have standalone unit tests?). And slightly harder things, like can the value based implementation layer assume it is never instantiated as a standalone value-based range? It sounds like more design decisions to achieve the goal.
Jun 22
prev sibling next sibling parent reply Avrina <avrina12309412342 gmail.com> writes:
On Saturday, 20 June 2020 at 14:51:32 UTC, Andrei Alexandrescu 
wrote:
 On 6/20/20 5:49 AM, Stanislav Blinov wrote:
 On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:
 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu 
 via Digitalmars-d wrote: [...]
 One good goal for std.v2020 would be to forego autodecoding 
 throughout.
[...] Another could be to fix up the range API -- i.e, reconsider the ugliness that is .save, now that D has copy ctors.
How do you see that? Fundamentally, what have copy ctors changed in this regard?
I've been thinking of this for a while. The fact that input ranges must buffer one element (i.e. make front() callable several times) has been a gauche choice. Input ranges should have only one API: bool fetchNext(T& target); Fill the user-provided target with the next element and return true. At the end of the range, return false and leave the target alone. I'd considered this design back in the day but I was annoyed that it required a copy for ranges that in fact are better than input ranges (e.g. arrays etc). It's still a disadvantage, but over the years I realized there are not that many algorithms that only work on input ranges so we could make-do. Of those for which efficiency is a concern, they could easily be specialized for forward ranges.
Most of the algorithms in Phobos don't call front() multiple times and they make a copy. It is really annoying as this prevents using non copyable types or those that need to be moved, eg similar to unique_ptr. What you are suggesting doesn't sound all that better, it just cements what is already common place in Phobos into the API. It's usually not algorithms that are concerned with efficiency, but rather the type. If you implement an algorithm that you think doesn't need to be efficient but it does need to be for the type I am using it with, I don't think that'd any better than the situation we already have.
Jun 20
next sibling parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 20 June 2020 at 17:02:56 UTC, Avrina wrote:

 Most of the algorithms in Phobos don't call front() multiple 
 times and they make a copy. It is really annoying as this 
 prevents using non copyable types or those that need to be 
 moved, eg similar to unique_ptr. What you are suggesting 
 doesn't sound all that better, it just cements what is already 
 common place in Phobos into the API.
On the contrary. Non-copyable types are *the* target for a hypothetical fetchNext(ref T target). Which would be system, and you'd be calling it with an uninitialized value. Although a one-pass consuming range could also be built with the `moveFront` interface. That said, some algorithms do indeed require an improvement (toward moving instead of copying). `sort` just recently got such improvement courtesy of MoonlightSentinel.
Jun 20
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/20 1:02 PM, Avrina wrote:
 Most of the algorithms in Phobos don't call front() multiple times and 
 they make a copy.
Interesting. They should be easy to fix then. Got a few examples?
Jun 20
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, June 20, 2020 6:07:29 PM MDT Andrei Alexandrescu via Digitalmars-
d wrote:
 On 6/20/20 1:02 PM, Avrina wrote:
 Most of the algorithms in Phobos don't call front() multiple times and
 they make a copy.
Interesting. They should be easy to fix then. Got a few examples?
Part of the problem is that sometimes, you very much want to be calling front only once and storing the result, because front is calculated every time that it's called (e.g. map calculates front on every call), whereas other times, you want to call front multiple times so that you can avoid copying the return value. To make matters worse, in the case of something like map, while you might get equivalent values on each call to front, you won't necessarily get exactly the same value with each call to front (e.g. if map's predicate calls new). So, there's a strong argument to be made that it's more correct and likely more efficient to call front only once even if that means copying the return value. However, AFAIK, there isn't really consensus on what the correct approach is for generic code, and I don't know what the more common approach is in Phobos. I suppose that code could do introspection on front to see whether it returned by ref such that it called front multiple times if it returned by ref and avoided calling it multiple times if it didn't, but I rather doubt that much of anyone is going to do something like that in practice. Obviously, if non-copyable types were to be supported, then it needs to be the norm to avoid copying the return value of front, but non-copyable types are a bit of a disaster with ranges anyway, since it's so trivial for them to result in the range itself being non-copyable (at which point, it doesn't even work with foreach), and it's very common for algorithms to assume that a range's elements are copyable. It also risks having to have code test for copyability rather than assuming that it's possible, which will add yet another stray ability that range-based code will have to test for. Personally, I don't think that supporting non-copyable types is worth it and that isInputRange should just ban them, but obviously, some folks want non-copyable types to be supported. However, isForwardRange does currently ban them (though I don't know if it's on purpose on just a happy accident). So, there's that. - Jonathan M Davis
Jun 22
prev sibling next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Saturday, 20 June 2020 at 14:51:32 UTC, Andrei Alexandrescu 
wrote:
 [snip]

 I've been thinking of this for a while. The fact that input 
 ranges must buffer one element (i.e. make front() callable 
 several times) has been a gauche choice.

 Input ranges should have only one API:

 bool fetchNext(T& target);

 Fill the user-provided target with the next element and return 
 true. At the end of the range, return false and leave the 
 target alone.

 I'd considered this design back in the day but I was annoyed 
 that it required a copy for ranges that in fact are better than 
 input ranges (e.g. arrays etc).

 It's still a disadvantage, but over the years I realized there 
 are not that many algorithms that only work on input ranges so 
 we could make-do. Of those for which efficiency is a concern, 
 they could easily be specialized for forward ranges.
That similar to what Steven Schveighoffer did in dcollections with cursors. https://github.com/schveiguy/dcollections/blob/master/concepts.txt
Jun 20
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/20/20 2:04 PM, jmh530 wrote:
 On Saturday, 20 June 2020 at 14:51:32 UTC, Andrei Alexandrescu wrote:
 [snip]

 I've been thinking of this for a while. The fact that input ranges 
 must buffer one element (i.e. make front() callable several times) has 
 been a gauche choice.

 Input ranges should have only one API:

 bool fetchNext(T& target);

 Fill the user-provided target with the next element and return true. 
 At the end of the range, return false and leave the target alone.

 I'd considered this design back in the day but I was annoyed that it 
 required a copy for ranges that in fact are better than input ranges 
 (e.g. arrays etc).

 It's still a disadvantage, but over the years I realized there are not 
 that many algorithms that only work on input ranges so we could 
 make-do. Of those for which efficiency is a concern, they could easily 
 be specialized for forward ranges.
That similar to what Steven Schveighoffer did in dcollections with cursors. https://github.com/schveiguy/dcollections/blob/master/concepts.txt
Sorry, this is not the same. Cursors are simply a way to refer to exactly one element, instead of using a begin and end element like a traditional container range would. The advantage is that removing the "end" element (that has nothing to do with the range) won't invalidate the cursor. What I wanted was the functionality of C++ iterators without the danger that comes with actually iterating them. -Steve
Jun 20
parent jmh530 <john.michael.hall gmail.com> writes:
On Saturday, 20 June 2020 at 18:33:46 UTC, Steven Schveighoffer 
wrote:
 [snip]

 Sorry, this is not the same.

 Cursors are simply a way to refer to exactly one element, 
 instead of using a begin and end element like a traditional 
 container range would.

 The advantage is that removing the "end" element (that has 
 nothing to do with the range) won't invalidate the cursor.

 What I wanted was the functionality of C++ iterators without 
 the danger that comes with actually iterating them.

 -Steve
Andrei's proposal was for a function that returns true and fills the referred to variable and returns false otherwise. Cursors are obviously different, but they have a similarity in that a simple cursor, like below, has true/false behavior by storing it in empty. struct Cursor(T) { T* ptr; bool empty = false; } That aspect of it was what I was thinking of. So for instance, if you have a struct that has a Cursor as a member, then you could do something like below. bool fetchNext(ref T target) { cursor.popFront; //need to define popFront target = cursor.ptr; return cursor.empty; }
Jun 20
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 20.06.20 16:51, Andrei Alexandrescu wrote:
 
 Input ranges should have only one API:
 
 bool fetchNext(T& target);
 
 Fill the user-provided target with the next element and return true. At 
 the end of the range, return false and leave the target alone.
What if the caller does not know how to construct a T? Nullable!T fetchNext(); (I get that D's support for algebraic data types is subpar, but maybe that is something to look into.)
Jun 21
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/21/20 6:06 PM, Timon Gehr wrote:
 On 20.06.20 16:51, Andrei Alexandrescu wrote:
 Input ranges should have only one API:

 bool fetchNext(T& target);

 Fill the user-provided target with the next element and return true. 
 At the end of the range, return false and leave the target alone.
What if the caller does not know how to construct a T? Nullable!T fetchNext(); (I get that D's support for algebraic data types is subpar, but maybe that is something to look into.)
That also creates a copy for each element, which may be disadvantageous for certain range types. (Also does not give access to references to the original elements, for you can't say Nullable!(ref T).) Making an API that is simple and works with in-memory elements as well as values synthesized on the fly is quite a challenge.
Jun 21
prev sibling next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:
 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu 
 via Digitalmars-d wrote: [...]
 One good goal for std.v2020 would be to forego autodecoding 
 throughout.
[...] Another could be to fix up the range API -- i.e, reconsider the ugliness that is .save, now that D has copy ctors. T
Also, switch from `void popFront()` to `typeof(this) rest`, so that we can have `const` and `immutable` ranges.
Jun 20
next sibling parent Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Saturday, 20 June 2020 at 10:43:41 UTC, Paul Backus wrote:
 On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:
 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu 
 via Digitalmars-d wrote: [...]
 One good goal for std.v2020 would be to forego autodecoding 
 throughout.
[...] Another could be to fix up the range API -- i.e, reconsider the ugliness that is .save, now that D has copy ctors. T
Also, switch from `void popFront()` to `typeof(this) rest`, so that we can have `const` and `immutable` ranges.
Yes!
Jun 20
prev sibling next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Saturday, 20 June 2020 at 10:43:41 UTC, Paul Backus wrote:
 [snip]

 Also, switch from `void popFront()` to `typeof(this) rest`, so 
 that we can have `const` and `immutable` ranges.
Would typeof(this) work where popFront is currently a free-standing function, like with built-in dynamic arrays?
Jun 20
parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Saturday, 20 June 2020 at 12:05:18 UTC, jmh530 wrote:
 On Saturday, 20 June 2020 at 10:43:41 UTC, Paul Backus wrote:
 [snip]

 Also, switch from `void popFront()` to `typeof(this) rest`, so 
 that we can have `const` and `immutable` ranges.
Would typeof(this) work where popFront is currently a free-standing function, like with built-in dynamic arrays?
typeof(this) is only possible for member function based implementations. Otherwise the implementation for arrays would be: inout(T)[] rest(T)(input(T)[] array) { return array[1 .. $]; } https://run.dlang.io/gist/run-dlang/7a4e5872ee26c083b916303fe36ac43b?compiler=dmd
Jun 20
parent jmh530 <john.michael.hall gmail.com> writes:
On Saturday, 20 June 2020 at 12:26:55 UTC, Petar Kirov 
[ZombineDev] wrote:
 [snip]

 typeof(this) is only possible for member function based 
 implementations. Otherwise the implementation for arrays would 
 be:

 inout(T)[] rest(T)(input(T)[] array) { return array[1 .. $]; }

 https://run.dlang.io/gist/run-dlang/7a4e5872ee26c083b916303fe36ac43b?compiler=dmd
Gotcha, thanks.
Jun 20
prev sibling next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 20 June 2020 at 10:43:41 UTC, Paul Backus wrote:

 Also, switch from `void popFront()` to `typeof(this) rest`, so 
 that we can have `const` and `immutable` ranges.
*Switch* is probably too restrictive. For a given range a `popFront` may be more efficient than a `range = range.rest`. Making `rest` a valid range primitive though, and using where appropriate - that'd be awesome.
Jun 20
next sibling parent Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Saturday, 20 June 2020 at 12:42:01 UTC, Stanislav Blinov wrote:
 On Saturday, 20 June 2020 at 10:43:41 UTC, Paul Backus wrote:

 Also, switch from `void popFront()` to `typeof(this) rest`, so 
 that we can have `const` and `immutable` ranges.
*Switch* is probably too restrictive. For a given range a `popFront` may be more efficient than a `range = range.rest`. Making `rest` a valid range primitive though, and using where appropriate - that'd be awesome.
Yes, that may be a valid concern, I have thought about this as well. The main issue is for some kinds of iterables, like alias sequences, we have no choice but to use an immutable-friendly API like rest/dropOne or concat (not append). A couple of years ago I was able to write proof-of-concept library that had accumulate, map, and filter implementations that worked with both ranges and template parameter sequences, based on immutable-friendly functions like this See: https://github.com/PetarKirov/rxd/blob/v0.0.3/source/rxd/xf/xform.d Some of the technics that I had developed allowed me to implement UFCS-enabled alternatives to std.meta that accept a polymorphic lambda parameter for mapping/filtering/etc.: https://gist.github.com/PetarKirov/a808c94857de84858accfb094c19bf77
Jun 20
prev sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Saturday, 20 June 2020 at 12:42:01 UTC, Stanislav Blinov wrote:
 On Saturday, 20 June 2020 at 10:43:41 UTC, Paul Backus wrote:

 Also, switch from `void popFront()` to `typeof(this) rest`, so 
 that we can have `const` and `immutable` ranges.
*Switch* is probably too restrictive. For a given range a `popFront` may be more efficient than a `range = range.rest`. Making `rest` a valid range primitive though, and using where appropriate - that'd be awesome.
For non-forward ranges, there's no promise that the original range (or any copies of it) will remain valid after calling `rest`, so you can always implement `rest` like this: auto rest() { this.popFront; return this; }
Jun 20
next sibling parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 20 June 2020 at 13:07:41 UTC, Paul Backus wrote:
 On Saturday, 20 June 2020 at 12:42:01 UTC, Stanislav Blinov 
 wrote:
 On Saturday, 20 June 2020 at 10:43:41 UTC, Paul Backus wrote:

 Also, switch from `void popFront()` to `typeof(this) rest`, 
 so that we can have `const` and `immutable` ranges.
*Switch* is probably too restrictive. For a given range a `popFront` may be more efficient than a `range = range.rest`. Making `rest` a valid range primitive though, and using where appropriate - that'd be awesome.
For non-forward ranges, there's no promise that the original range (or any copies of it) will remain valid after calling `rest`, so you can always implement `rest` like this: auto rest() { this.popFront; return this; }
For sure. I'm just pointing out that *switching* to `rest` isn't a likely way to go. I.e. that would've implied dropping the use of popFront altogether :) *Adopting* `rest` - yes, +100%.
Jun 20
parent Paul Backus <snarwin gmail.com> writes:
On Saturday, 20 June 2020 at 13:29:43 UTC, Stanislav Blinov wrote:
 On Saturday, 20 June 2020 at 13:07:41 UTC, Paul Backus wrote:
 On Saturday, 20 June 2020 at 12:42:01 UTC, Stanislav Blinov 
 wrote:
 On Saturday, 20 June 2020 at 10:43:41 UTC, Paul Backus wrote:

 Also, switch from `void popFront()` to `typeof(this) rest`, 
 so that we can have `const` and `immutable` ranges.
*Switch* is probably too restrictive. For a given range a `popFront` may be more efficient than a `range = range.rest`. Making `rest` a valid range primitive though, and using where appropriate - that'd be awesome.
For non-forward ranges, there's no promise that the original range (or any copies of it) will remain valid after calling `rest`, so you can always implement `rest` like this: auto rest() { this.popFront; return this; }
For sure. I'm just pointing out that *switching* to `rest` isn't a likely way to go. I.e. that would've implied dropping the use of popFront altogether :) *Adopting* `rest` - yes, +100%.
By "switching", I just mean that `std.v2.range.isInputRange` would only check for `rest`, not `popFront`, and that new code would be free to only implement `rest`. Naturally, we would also provide free-function versions of `rest` and `popFront` for compatibility between the two APIs.
Jun 20
prev sibling parent reply Johannes Loher <johannes.loher fg4f.de> writes:
On Saturday, 20 June 2020 at 13:07:41 UTC, Paul Backus wrote:
 On Saturday, 20 June 2020 at 12:42:01 UTC, Stanislav Blinov 
 wrote:
 On Saturday, 20 June 2020 at 10:43:41 UTC, Paul Backus wrote:

 Also, switch from `void popFront()` to `typeof(this) rest`, 
 so that we can have `const` and `immutable` ranges.
*Switch* is probably too restrictive. For a given range a `popFront` may be more efficient than a `range = range.rest`. Making `rest` a valid range primitive though, and using where appropriate - that'd be awesome.
For non-forward ranges, there's no promise that the original range (or any copies of it) will remain valid after calling `rest`, so you can always implement `rest` like this: auto rest() { this.popFront; return this; }
Can’t make it const then... also some ranges probably cannot bee const by design, e.g. byLine etc.
Jun 20
next sibling parent Paul Backus <snarwin gmail.com> writes:
On Saturday, 20 June 2020 at 18:34:37 UTC, Johannes Loher wrote:
 On Saturday, 20 June 2020 at 13:07:41 UTC, Paul Backus wrote:
 For non-forward ranges, there's no promise that the original 
 range (or any copies of it) will remain valid after calling 
 `rest`, so you can always implement `rest` like this:

     auto rest() {
         this.popFront;
         return this;
     }
Can’t make it const then... also some ranges probably cannot bee const by design, e.g. byLine etc.
Yes, ranges with popFront can't be const. The difference is that, with `rest`/`tail` as the default, *some* ranges that are currently forced to use popFront unnecessarily can be made const-compatible. Ranges that continue to use popFront, for whatever reason (performance, backwards compatibility, etc.) will remain mutable-only.
Jun 20
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/20 2:34 PM, Johannes Loher wrote:
 On Saturday, 20 June 2020 at 13:07:41 UTC, Paul Backus wrote:
 On Saturday, 20 June 2020 at 12:42:01 UTC, Stanislav Blinov wrote:
 On Saturday, 20 June 2020 at 10:43:41 UTC, Paul Backus wrote:

 Also, switch from `void popFront()` to `typeof(this) rest`, so that 
 we can have `const` and `immutable` ranges.
*Switch* is probably too restrictive. For a given range a `popFront` may be more efficient than a `range = range.rest`. Making `rest` a valid range primitive though, and using where appropriate - that'd be awesome.
For non-forward ranges, there's no promise that the original range (or any copies of it) will remain valid after calling `rest`, so you can always implement `rest` like this:     auto rest() {         this.popFront;         return this;     }
Can’t make it const then... also some ranges probably cannot bee const by design, e.g. byLine etc.
That function would be a fallback. Ranges that are const would implement it as a primitive not expressible with popFront. Consider: immutable List(T) { private T payload; private List next; public List tail() { return next; } ... } You can't implement popFront for this structure.
Jun 20
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/20 6:43 AM, Paul Backus wrote:
 On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:
 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu via 
 Digitalmars-d wrote: [...]
 One good goal for std.v2020 would be to forego autodecoding throughout.
[...] Another could be to fix up the range API -- i.e, reconsider the ugliness that is .save, now that D has copy ctors. T
Also, switch from `void popFront()` to `typeof(this) rest`, so that we can have `const` and `immutable` ranges.
Yah, tail() would be necessary too. popFront needs to stay because it doesn't copy the range.
Jun 20
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/20/20 6:43 AM, Paul Backus wrote:
 On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:
 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu via 
 Digitalmars-d wrote: [...]
 One good goal for std.v2020 would be to forego autodecoding throughout.
[...] Another could be to fix up the range API -- i.e, reconsider the ugliness that is .save, now that D has copy ctors. T
Also, switch from `void popFront()` to `typeof(this) rest`, so that we can have `const` and `immutable` ranges.
How does that work? You'd have to use recursion I guess? ugly. Why do we need ranges for something like this? -Steve
Jun 20
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/20 3:03 PM, Steven Schveighoffer wrote:
 On 6/20/20 6:43 AM, Paul Backus wrote:
 On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:
 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu via 
 Digitalmars-d wrote: [...]
 One good goal for std.v2020 would be to forego autodecoding throughout.
[...] Another could be to fix up the range API -- i.e, reconsider the ugliness that is .save, now that D has copy ctors. T
Also, switch from `void popFront()` to `typeof(this) rest`, so that we can have `const` and `immutable` ranges.
How does that work? You'd have to use recursion I guess? ugly. Why do we need ranges for something like this?
Think Hakell lists. They can implement tail easily (just return the next pointer) but can't define popFront.
Jun 20
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/20/20 8:12 PM, Andrei Alexandrescu wrote:
 On 6/20/20 3:03 PM, Steven Schveighoffer wrote:
 On 6/20/20 6:43 AM, Paul Backus wrote:
 On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:
 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu via 
 Digitalmars-d wrote: [...]
 One good goal for std.v2020 would be to forego autodecoding 
 throughout.
[...] Another could be to fix up the range API -- i.e, reconsider the ugliness that is .save, now that D has copy ctors.
Also, switch from `void popFront()` to `typeof(this) rest`, so that we can have `const` and `immutable` ranges.
How does that work? You'd have to use recursion I guess? ugly. Why do we need ranges for something like this?
Think Hakell lists. They can implement tail easily (just return the next pointer) but can't define popFront.
My question wasn't about how such a thing could be implemented, but how it works with const ranges. foreach(x; someConstRange) I think wouldn't be possible. I think you'd have to recurse: void process(const Range r) { subProcess(r.front); process(r.rest); } The point is to question the statement "so that we can have `const` and `immutable` ranges". Sure, we could implement recursive versions of find, etc. I don't know if that's worth it. -Steve
Jun 22
next sibling parent reply FeepingCreature <feepingcreature gmail.com> writes:
On Monday, 22 June 2020 at 12:15:39 UTC, Steven Schveighoffer 
wrote:
 On 6/20/20 8:12 PM, Andrei Alexandrescu wrote:
 On 6/20/20 3:03 PM, Steven Schveighoffer wrote:
 On 6/20/20 6:43 AM, Paul Backus wrote:
 On Saturday, 20 June 2020 at 04:34:42 UTC, H. S. Teoh wrote:
 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei 
 Alexandrescu via Digitalmars-d wrote: [...]
 One good goal for std.v2020 would be to forego 
 autodecoding throughout.
[...] Another could be to fix up the range API -- i.e, reconsider the ugliness that is .save, now that D has copy ctors.
Also, switch from `void popFront()` to `typeof(this) rest`, so that we can have `const` and `immutable` ranges.
How does that work? You'd have to use recursion I guess? ugly. Why do we need ranges for something like this?
Think Hakell lists. They can implement tail easily (just return the next pointer) but can't define popFront.
My question wasn't about how such a thing could be implemented, but how it works with const ranges. foreach(x; someConstRange) I think wouldn't be possible. I think you'd have to recurse: void process(const Range r) { subProcess(r.front); process(r.rest); }
I think maybe what is needed is a notion of "head-mutable"? void process(const headmut Range r) { while (!r.empty) { // subProcess still has to take an immmutable r. this does not expose mutable state. // headmut converts to immutable on pass-by-value because it makes a copy. subProcess(r.front); r = r.rest; // can be assigned because headmut, meaning references are still immutable } } I don't know how this would interact with dip1000. Maybe headmut could convert as immutable but expression scoped, which would come down to the same thing. Something like this is also probably the right backing type to use for many containers, like Nullable.
Jun 22
parent reply Seb <seb wilzba.ch> writes:
On Monday, 22 June 2020 at 12:28:06 UTC, FeepingCreature wrote:
 I think maybe what is needed is a notion of "head-mutable"?
It's called Final and already in Phobos: https://dlang.org/phobos/std_experimental_typecons.html#.Final Though it comes with a few bugs like e.g. [1] or [2]. [1] https://issues.dlang.org/show_bug.cgi?id=17272 [2] https://github.com/dlang/phobos/pull/6596
Jun 22
parent reply Seb <seb wilzba.ch> writes:
On Monday, 22 June 2020 at 15:29:25 UTC, Seb wrote:
 On Monday, 22 June 2020 at 12:28:06 UTC, FeepingCreature wrote:
 I think maybe what is needed is a notion of "head-mutable"?
It's called Final and already in Phobos: https://dlang.org/phobos/std_experimental_typecons.html#.Final Though it comes with a few bugs like e.g. [1] or [2]. [1] https://issues.dlang.org/show_bug.cgi?id=17272 [2] https://github.com/dlang/phobos/pull/6596
Sorry. I read to quick. Final is head-const and tail-mutable. Rebindable is the Phobos solution to head-mutable / tail-const: https://dlang.org/phobos/std_typecons.html#Rebindable The biggest gotcha of Rebindable is that it doesn't work with structs [1] which vastly limits its usability. [1] https://github.com/dlang/phobos/pull/6136
Jun 22
parent FeepingCreature <feepingcreature gmail.com> writes:
On Monday, 22 June 2020 at 15:34:36 UTC, Seb wrote:
 The biggest gotcha of Rebindable is that it doesn't work with 
 structs [1] which vastly limits its usability.

 [1] https://github.com/dlang/phobos/pull/6136
Yes, effectively what I'm looking for is "Rebindable for immutable structs." I already have this in Nullable, but it's an atrocious hack because I basically have to hit the compiler on the head until it forgets what const is. I want some way to say "yes, this type is immutable, but it's only immutable over its lifetime, and *I* control its lifetime and *I* say when it ends - and when it ends, I'm free to put other data there, such as the return value of rest()." This is also vital for finally allowing immutable types in (key-)mutable hashmaps.
Jun 23
prev sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 22 June 2020 at 12:15:39 UTC, Steven Schveighoffer 
wrote:
 My question wasn't about how such a thing could be implemented, 
 but how it works with const ranges.

 foreach(x; someConstRange) I think wouldn't be possible. I 
 think you'd have to recurse:

 void process(const Range r)
 {
    subProcess(r.front);
    process(r.rest);
 }

 The point is to question the statement "so that we can have 
 `const` and `immutable` ranges".

 Sure, we could implement recursive versions of find, etc. I 
 don't know if that's worth it.

 -Steve
Well, currently, range algorithms can't work with const ranges *at all*, recursively or iteratively. So from a user perspective, this would be a strict improvement on the status quo.
Jun 22
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/22/20 10:20 AM, Paul Backus wrote:
 On Monday, 22 June 2020 at 12:15:39 UTC, Steven Schveighoffer wrote:
 My question wasn't about how such a thing could be implemented, but 
 how it works with const ranges.

 foreach(x; someConstRange) I think wouldn't be possible. I think you'd 
 have to recurse:

 void process(const Range r)
 {
    subProcess(r.front);
    process(r.rest);
 }

 The point is to question the statement "so that we can have `const` 
 and `immutable` ranges".

 Sure, we could implement recursive versions of find, etc. I don't know 
 if that's worth it.
Well, currently, range algorithms can't work with const ranges *at all*, recursively or iteratively. So from a user perspective, this would be a strict improvement on the status quo.
Algorithms can work with const ranges -- as long as the range is an array: const int[] arr = [1, 2, 3, 4, 5]; auto x = arr.find(3); assert(x == [3, 4, 5]); I think the better option is to focus on making it possible to duplicate this possibility for generic ranges rather than implement a new and awkward API. FeepingCreature is right, we should try and create a head mutable mechanism. I have envisioned it in the past as a tail-modifier mechanism (e.g. tail-const). -Steve
Jun 22
next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 22 June 2020 at 14:33:53 UTC, Steven Schveighoffer 
wrote:
 On 6/22/20 10:20 AM, Paul Backus wrote:
 
 Well, currently, range algorithms can't work with const ranges 
 *at all*, recursively or iteratively. So from a user 
 perspective, this would be a strict improvement on the status 
 quo.
Algorithms can work with const ranges -- as long as the range is an array: const int[] arr = [1, 2, 3, 4, 5]; auto x = arr.find(3); assert(x == [3, 4, 5]); I think the better option is to focus on making it possible to duplicate this possibility for generic ranges rather than implement a new and awkward API. FeepingCreature is right, we should try and create a head mutable mechanism. I have envisioned it in the past as a tail-modifier mechanism (e.g. tail-const). -Steve
This isn't really "algorithms working with const ranges"; rather, it's "const(T[]) implicitly converts to const(T)[]". The algorithm itself never sees a const range. I suppose we could have the same thing for user-defined types if we added ` implicit opCast`, or a more restrictive version like `opHeadMutable`, but the whole thing seems backwards to me. const(T[]) converting to const(T)[] is a special case in the language introduced to compensate for the shortcomings `popFront`. Rather than doubling down on it, why not fix the issue at its source?
Jun 22
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/22/20 10:55 AM, Paul Backus wrote:
 On Monday, 22 June 2020 at 14:33:53 UTC, Steven Schveighoffer wrote:
 On 6/22/20 10:20 AM, Paul Backus wrote:
 Well, currently, range algorithms can't work with const ranges *at 
 all*, recursively or iteratively. So from a user perspective, this 
 would be a strict improvement on the status quo.
Algorithms can work with const ranges -- as long as the range is an array: const int[] arr = [1, 2, 3, 4, 5]; auto x = arr.find(3); assert(x == [3, 4, 5]); I think the better option is to focus on making it possible to duplicate this possibility for generic ranges rather than implement a new and awkward API. FeepingCreature is right, we should try and create a head mutable mechanism. I have envisioned it in the past as a tail-modifier mechanism (e.g. tail-const).
This isn't really "algorithms working with const ranges"; rather, it's "const(T[]) implicitly converts to const(T)[]". The algorithm itself never sees a const range.
I don't see a difference. When you copy a range as a parameter, the head is a different piece of memory. This is why it works. Why is it important how it works?
 I suppose we could have the same thing for user-defined types if we 
 added ` implicit opCast`, or a more restrictive version like 
 `opHeadMutable`, but the whole thing seems backwards to me. const(T[]) 
 converting to const(T)[] is a special case in the language introduced to 
 compensate for the shortcomings `popFront`. Rather than doubling down on 
 it, why not fix the issue at its source?
That isn't why tail-const was "introduced". And I would consider enabling this feature for custom types fixing it at the source. Why does it seem backwards to you? Perhaps you are thinking of how templates automatically strip the head const? That I don't necessarily agree with either, but there isn't a good way to say "take this parameter as head-mutable" in generic code, which is why *that* was added. If we did have a mechanism to say that (such as FeepingCreture's `headmut` example), then we wouldn't need that special treatment. -Steve
Jun 22
parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 22 June 2020 at 15:47:51 UTC, Steven Schveighoffer 
wrote:
 On 6/22/20 10:55 AM, Paul Backus wrote:
 
 This isn't really "algorithms working with const ranges"; 
 rather, it's "const(T[]) implicitly converts to const(T)[]". 
 The algorithm itself never sees a const range.
I don't see a difference. When you copy a range as a parameter, the head is a different piece of memory. This is why it works. Why is it important how it works?
It's important how it works because it *doesn't* work, for ranges in general. It only works for slices. Making it work in general requires either subverting the type system (like Rebindable does) or adding more special cases to the language.
 Perhaps you are thinking of how templates automatically strip 
 the head const? That I don't necessarily agree with either, but 
 there isn't a good way to say "take this parameter as 
 head-mutable" in generic code, which is why *that* was added. 
 If we did have a mechanism to say that (such as 
 FeepingCreture's `headmut` example), then we wouldn't need that 
 special treatment.
The question is, why do you need a mechanism to say that in the first place? Why not just have generic code take the argument as its actual type? Answer: because popFront requires a mutable object. `headmut` is a solution to a problem that doesn't need to exist in the first place. Get rid of popFront, and you get rid of the problem.
Jun 22
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 6/22/20 12:14 PM, Paul Backus wrote:
 Making it work in general requires either subverting the type system 
 (like Rebindable does) or adding more special cases to the language.
The principled approach is to generalize the trick currently used only for arrays. Consider the lowering: "Whenever something is passed to a function, .opOnCall is appended." Then: auto ref T opOnCall(T)(auto ref T x) { return x; } const(T)[] opOnCall(T)(const(T[]) x) { return x; } immutable(T)[] opOnCall(T)(immutable(T[]) x) { return x; } This is just a sketch, haven't thought it through, but if properly defined the scheme would work having built-in arrays work as they are today, and would also allow user-defined types to define their own opOnCall as they find fit.
Jun 22
parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 22 June 2020 at 16:42:55 UTC, Andrei Alexandrescu 
wrote:
 On 6/22/20 12:14 PM, Paul Backus wrote:
 Making it work in general requires either subverting the type 
 system (like Rebindable does) or adding more special cases to 
 the language.
The principled approach is to generalize the trick currently used only for arrays. Consider the lowering: "Whenever something is passed to a function, .opOnCall is appended." Then: auto ref T opOnCall(T)(auto ref T x) { return x; } const(T)[] opOnCall(T)(const(T[]) x) { return x; } immutable(T)[] opOnCall(T)(immutable(T[]) x) { return x; } This is just a sketch, haven't thought it through, but if properly defined the scheme would work having built-in arrays work as they are today, and would also allow user-defined types to define their own opOnCall as they find fit.
The trick used for arrays does not only apply to function calls: const(int[]) a = [1, 2, 3]; const(int)[] b = a; // compiles IMHO the principled way to allow user-defined implicit conversions is...to allow user-defined implicit conversions. But iirc that's a can of worms Walter prefers not to open.
Jun 22
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 6/22/20 12:50 PM, Paul Backus wrote:
 On Monday, 22 June 2020 at 16:42:55 UTC, Andrei Alexandrescu wrote:
 On 6/22/20 12:14 PM, Paul Backus wrote:
 Making it work in general requires either subverting the type system 
 (like Rebindable does) or adding more special cases to the language.
The principled approach is to generalize the trick currently used only for arrays. Consider the lowering: "Whenever something is passed to a function, .opOnCall is appended." Then: auto ref T opOnCall(T)(auto ref T x) { return x; } const(T)[] opOnCall(T)(const(T[]) x) { return x; } immutable(T)[] opOnCall(T)(immutable(T[]) x) { return x; } This is just a sketch, haven't thought it through, but if properly defined the scheme would work having built-in arrays work as they are today, and would also allow user-defined types to define their own opOnCall as they find fit.
The trick used for arrays does not only apply to function calls:     const(int[]) a = [1, 2, 3];     const(int)[] b = a; // compiles
That's different - it's an implicit conversion.
 IMHO the principled way to allow user-defined implicit conversions 
 is...to allow user-defined implicit conversions. But iirc that's a can 
 of worms Walter prefers not to open.
What happens upon function calls is not an implicit conversion. It's a forced type change.
Jun 22
next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 22 June 2020 at 21:46:51 UTC, Andrei Alexandrescu 
wrote:
 On 6/22/20 12:50 PM, Paul Backus wrote:
 
 The trick used for arrays does not only apply to function 
 calls:
 
      const(int[]) a = [1, 2, 3];
      const(int)[] b = a; // compiles
That's different - it's an implicit conversion.
 IMHO the principled way to allow user-defined implicit 
 conversions is...to allow user-defined implicit conversions. 
 But iirc that's a can of worms Walter prefers not to open.
What happens upon function calls is not an implicit conversion. It's a forced type change.
So what you're saying is, it's even *less* principled than I thought? :) Regardless, my broader point is that once we're open to the possibility of designing a new range API, all of this can be solved without any language changes by using an API that doesn't require mutation (i.e., tail()). Surely that's a better solution than implicitly inserting calls to arbitrary user-defined code every time someone passes an argument to a function.
Jun 22
next sibling parent reply Aliak <something something.com> writes:
On Monday, 22 June 2020 at 22:17:43 UTC, Paul Backus wrote:
 On Monday, 22 June 2020 at 21:46:51 UTC, Andrei Alexandrescu 
 wrote:
 On 6/22/20 12:50 PM, Paul Backus wrote:
 
 The trick used for arrays does not only apply to function 
 calls:
 
      const(int[]) a = [1, 2, 3];
      const(int)[] b = a; // compiles
That's different - it's an implicit conversion.
 IMHO the principled way to allow user-defined implicit 
 conversions is...to allow user-defined implicit conversions. 
 But iirc that's a can of worms Walter prefers not to open.
What happens upon function calls is not an implicit conversion. It's a forced type change.
So what you're saying is, it's even *less* principled than I thought? :)
Have you guys seen the work done here: https://gist.github.com/Biotronic/940f553fa1b4cbf80a98afe3d1970c6d ? It is essentially the opOnCall with details on the library implementation as well.
 Regardless, my broader point is that once we're open to the 
 possibility of designing a new range API, all of this can be 
 solved without any language changes by using an API that 
 doesn't require mutation (i.e., tail()). Surely that's a better 
 solution than implicitly inserting calls to arbitrary 
 user-defined code every time someone passes an argument to a 
 function.
Can you use the tail() approach and keep the efficiency of popFront? - I.e. do you see that if tail is const it returns a newly constructed range and if tail() is mutable it can internally call a popFront links think and return a ref to itself?
Jun 22
parent Paul Backus <snarwin gmail.com> writes:
On Monday, 22 June 2020 at 22:36:05 UTC, Aliak wrote:
 On Monday, 22 June 2020 at 22:17:43 UTC, Paul Backus wrote:
 Regardless, my broader point is that once we're open to the 
 possibility of designing a new range API, all of this can be 
 solved without any language changes by using an API that 
 doesn't require mutation (i.e., tail()). Surely that's a 
 better solution than implicitly inserting calls to arbitrary 
 user-defined code every time someone passes an argument to a 
 function.
Can you use the tail() approach and keep the efficiency of popFront? - I.e. do you see that if tail is const it returns a newly constructed range and if tail() is mutable it can internally call a popFront links think and return a ref to itself?
In a world where tail() is a range primitive, popFront becomes an algorithm: void popFront(R)(ref R r) if (isInputRangeV2!R && isMutable!R) { r = r.tail; } Any decent optimizer should be able to turn this into a mutation. Since ranges already rely heavily on optimization to produce efficient code, I don't think it's unreasonable to rely on it here as well. Of course, if you need more control, you are always free to specialize the algorithm for your own range type by defining a popFront method.
Jun 22
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/22/20 6:17 PM, Paul Backus wrote:
 On Monday, 22 June 2020 at 21:46:51 UTC, Andrei Alexandrescu wrote:
 On 6/22/20 12:50 PM, Paul Backus wrote:
 The trick used for arrays does not only apply to function calls:

      const(int[]) a = [1, 2, 3];
      const(int)[] b = a; // compiles
That's different - it's an implicit conversion.
 IMHO the principled way to allow user-defined implicit conversions 
 is...to allow user-defined implicit conversions. But iirc that's a 
 can of worms Walter prefers not to open.
What happens upon function calls is not an implicit conversion. It's a forced type change.
So what you're saying is, it's even *less* principled than I thought? :)
Yeah, it's a hack, but it works very well.
 Regardless, my broader point is that once we're open to the possibility 
 of designing a new range API, all of this can be solved without any 
 language changes by using an API that doesn't require mutation (i.e., 
 tail()).
Using only tail() makes iteration with mutable ranges inefficient and iteration with immutable ranges difficult (must use recursion everywhere). Hardly a winner of a design contest.
  Surely that's a better solution than implicitly inserting calls 
 to arbitrary user-defined code every time someone passes an argument to 
 a function.
If there is, we haven't found it.
Jun 22
parent reply Paul Backus <snarwin gmail.com> writes:
On Tuesday, 23 June 2020 at 05:15:49 UTC, Andrei Alexandrescu 
wrote:
 Regardless, my broader point is that once we're open to the 
 possibility of designing a new range API, all of this can be 
 solved without any language changes by using an API that 
 doesn't require mutation (i.e., tail()).
Using only tail() makes iteration with mutable ranges inefficient
I don't think it's a given that tail() is less efficient than popFront(). Here's a program where both tail() and popFront() produce the same assembly with `ldc -O2`: https://d.godbolt.org/z/-S_JoF Of course, if it does turn out to make a difference in some cases, mutable ranges are free to implement popFront as well. There will be a generic version in std.v2.algorithm to ensure that `mutableRange.popFront()` is always valid.
 and iteration with immutable ranges difficult (must use 
 recursion everywhere). Hardly a winner of a design contest.
Even if we assume for the sake of argument that "recursion == difficult", I would still say that "difficult" is an improvement over "impossible".
Jun 23
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/23/20 9:07 AM, Paul Backus wrote:
 On Tuesday, 23 June 2020 at 05:15:49 UTC, Andrei Alexandrescu wrote:
 Regardless, my broader point is that once we're open to the 
 possibility of designing a new range API, all of this can be solved 
 without any language changes by using an API that doesn't require 
 mutation (i.e., tail()).
Using only tail() makes iteration with mutable ranges inefficient
I don't think it's a given that tail() is less efficient than popFront(). Here's a program where both tail() and popFront() produce the same assembly with `ldc -O2`: https://d.godbolt.org/z/-S_JoF
With arrays, no problem. Two words to copy, easy to track, enregister etc. With elaborate ranges, definitely a problem.
 Of course, if it does turn out to make a difference in some cases, 
 mutable ranges are free to implement popFront as well. There will be a 
 generic version in std.v2.algorithm to ensure that 
 `mutableRange.popFront()` is always valid.
 
 and iteration with immutable ranges difficult (must use recursion 
 everywhere). Hardly a winner of a design contest.
Even if we assume for the sake of argument that "recursion == difficult", I would still say that "difficult" is an improvement over "impossible".
There's "onerous" somewhere in there, too. I agree with Steve who said implementing recursive variants of most range algorithms in Phobos would be onerous. The each() primitive I discussed may work nice actually because it allows recursion to be done in only itself, instead of every algorithm.
Jun 23
next sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Tuesday, 23 June 2020 at 16:20:26 UTC, Andrei Alexandrescu 
wrote:
 On 6/23/20 9:07 AM, Paul Backus wrote:
 I don't think it's a given that tail() is less efficient than 
 popFront(). Here's a program where both tail() and popFront() 
 produce the same assembly with `ldc -O2`:
 
 https://d.godbolt.org/z/-S_JoF
With arrays, no problem. Two words to copy, easy to track, enregister etc. With elaborate ranges, definitely a problem.
Do you (or anyone else reading this--feel free to chime in) have an example in mind of such an "elaborate range"? If so, I'd be happy to do the legwork of running additional experiments. No need to settle for speculation here when we can have actual data.
 Even if we assume for the sake of argument that "recursion == 
 difficult", I would still say that "difficult" is an 
 improvement over "impossible".
There's "onerous" somewhere in there, too. I agree with Steve who said implementing recursive variants of most range algorithms in Phobos would be onerous.
Again, if you, Steve, or anyone else in this thread have any examples in mind, I'd be happy to run the experiment of writing recursive versions myself.
 The each() primitive I discussed may work nice actually because 
 it allows recursion to be done in only itself, instead of every 
 algorithm.
Even without making `each` a primitive, you can do this. Implement a few fundamental algorithms like `each`, `map`, `filter`, and `fold` using recursion, and many others can be built on top of them.
Jun 23
next sibling parent Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Tuesday, 23 June 2020 at 21:19:39 UTC, Paul Backus wrote:
 Even without making `each` a primitive, you can do this. 
 Implement a few fundamental algorithms like `each`, `map`, 
 `filter`, and `fold` using recursion, and many others can be 
 built on top of them.
Slightly off-topic, but are you familiar with Clojure's transducers? https://clojure.org/reference/transducers https://www.youtube.com/watch?v=6mTbuzafcII You can also check this talk from CppCon, to see how their applied to a statically typed language: https://www.youtube.com/watch?v=6mTbuzafcII For me, the main advantage is that this design allows implementing algorithms like map, filter, etc. without caring at all about the exact range interface. This for me is a big deal because, in my day-to-day job, I can't use much range/iterator/iterable/enumerable pull-style algorithms, because most my code is reactive/ asynchronous (push-based). So most of what I do is based on either Futures/Promises or Reactive Extensions: http://reactivex.io/ http://reactivex.io/documentation/operators.html
Jun 24
prev sibling parent reply Jon Degenhardt <jond noreply.com> writes:
On Tuesday, 23 June 2020 at 21:19:39 UTC, Paul Backus wrote:
 Do you (or anyone else reading this--feel free to chime in) 
 have an example in mind of such an "elaborate range"? If so, 
 I'd be happy to do the legwork of running additional 
 experiments. No need to settle for speculation here when we can 
 have actual data.
I don't think they'll satisfy what you are looking for, but I do have a few examples of ranges doing non-trivial things in public repos. You are welcome to take a look: * bufferedByLine - https://github.com/eBay/tsv-utils/blob/master/common/src/tsv_utils/common/utils.d#L831 * inputSourceRange - https://github.com/eBay/tsv-utils/blob/master/common/src/tsv_utils/common/utils.d#L1443 * parseFieldList - https://github.com/eBay/tsv-utils/blob/master/common/src/tsv_utils/common/fieldlist.d#L291 * findFieldGroups - https://github.com/eBay/tsv-utils/blob/master/common/src/tsv_utils/common/fieldlist.d#L1045 * parseNumericFieldList - https://github.com/eBay/tsv-utils/blob/master/common/src/tsv_utils/common/fieldlist.d#L2029 None of them are used in performance sensitive regions of code, so they haven't been benchmarked or optimized.
Jun 24
parent Jon Degenhardt <jond noreply.com> writes:
On Wednesday, 24 June 2020 at 17:00:37 UTC, Jon Degenhardt wrote:
 On Tuesday, 23 June 2020 at 21:19:39 UTC, Paul Backus wrote:
 Do you (or anyone else reading this--feel free to chime in) 
 have an example in mind of such an "elaborate range"? If so, 
 I'd be happy to do the legwork of running additional 
 experiments. No need to settle for speculation here when we 
 can have actual data.
I don't think they'll satisfy what you are looking for, but I do have a few examples of ranges doing non-trivial things in public repos. You are welcome to take a look: * bufferedByLine - https://github.com/eBay/tsv-utils/blob/master/common/src/tsv_utils/common/utils.d#L831 * inputSourceRange - https://github.com/eBay/tsv-utils/blob/master/common/src/tsv_utils/common/utils.d#L1443 * parseFieldList - https://github.com/eBay/tsv-utils/blob/master/common/src/tsv_utils/common/fieldlist.d#L291 * findFieldGroups - https://github.com/eBay/tsv-utils/blob/master/common/src/tsv_utils/common/fieldlist.d#L1045 * parseNumericFieldList - https://github.com/eBay/tsv-utils/blob/master/common/src/tsv_utils/common/fieldlist.d#L2029 None of them are used in performance sensitive regions of code, so they haven't been benchmarked or optimized.
A correction to the last entry. The line number in the URL is to a different range. Still a valid range to look at, but in the end it just returns a Phobos range. The one I meant to include is a more typical range. The correct entries, replacing the last one: * parseNumericFieldGroup - https://github.com/eBay/tsv-utils/blob/master/common/src/tsv_utils/common/fieldlist.d#L2029 * parseNumericFieldList - https://github.com/eBay/tsv-utils/blob/master/common/src/tsv_utils/common/fieldlist.d#L2482
Jun 26
prev sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Tuesday, 23 June 2020 at 16:20:26 UTC, Andrei Alexandrescu 
wrote:
 On 6/23/20 9:07 AM, Paul Backus wrote:
 I don't think it's a given that tail() is less efficient than 
 popFront(). Here's a program where both tail() and popFront() 
 produce the same assembly with `ldc -O2`:
 
 https://d.godbolt.org/z/-S_JoF
With arrays, no problem. Two words to copy, easy to track, enregister etc. With elaborate ranges, definitely a problem. ... There's "onerous" somewhere in there, too. I agree with Steve who said implementing recursive variants of most range algorithms in Phobos would be onerous. The each() primitive I discussed may work nice actually because it allows recursion to be done in only itself, instead of every algorithm.
I'm struck that no one AFAICS has suggested the following alternative: instead of `tail`, to allow popFront() (and if implemented, popBack()) to return an `Option!(ElementType, None)`. What is returned would be either the popped element, or None if no more elements remain (with the nice by-product of no longer getting assert failures if pop{Front,Back} are called when the range is empty). This might allow some natural API evolution along the following lines: * an input range need only implement such an option-based popFront() and no other methods - for backwards compatibility, we could also support implementing `front` and a `void`-returning popFront() * a forward range would additionally implement `front` and `save` methods: - `front` because, since a forward range is deterministic, it should always be possible to know up front (pun intended...) what the first element should be - `save` for the usual reasons * all other range types should derive naturally from this - we might want to consider a more fine-grained approach to bidirectional ranges, e.g. allowing a bidirectional input range with option-returning popFront and popBack * over time we could/should deprecate void-returning pop{Front,Back}, and possibly also defining `front` or `back` if `save` is not also defined The approach of having only option-returning popFront would be particularly useful for input ranges where the front cannot or should not be predefined on construction (e.g. RNGs, or stuff that wraps them like RandomSample and friends). (Is there actually a good example of an input range where the first element can be predefined on construction in a way that isn't -- like pseudo-RNGs -- an implementation detail that one probably wants to hide from the user?)
Jun 24
next sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 24 June 2020 at 14:17:17 UTC, Joseph Rushton 
Wakeling wrote:
 [snip]

 I'm struck that no one AFAICS has suggested the following 
 alternative: instead of `tail`, to allow popFront() (and if 
 implemented, popBack()) to return an `Option!(ElementType, 
 None)`.

 [snip]
This is similar to what I had said previously [1] about cursors (though Steven disagreed with my initial take). However, it sounds like an Option is more elegant. [1] https://forum.dlang.org/post/nudkmmtoeqtznuorneev forum.dlang.org
Jun 24
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/24/20 10:17 AM, Joseph Rushton Wakeling wrote:
 
 I'm struck that no one AFAICS has suggested the following alternative: 
 instead of `tail`, to allow popFront() (and if implemented, popBack()) 
 to return an `Option!(ElementType, None)`.
 
 What is returned would be either the popped element, or None if no more 
 elements remain (with the nice by-product of no longer getting assert 
 failures if pop{Front,Back} are called when the range is empty).
I think the discussion about tail() had to do with immutable ranges. Having popXxx() return something would still have it mutate the range.
Jun 24
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Wednesday, 24 June 2020 at 16:08:05 UTC, Andrei Alexandrescu 
wrote:
 On 6/24/20 10:17 AM, Joseph Rushton Wakeling wrote:
 I'm struck that no one AFAICS has suggested the following 
 alternative: instead of `tail`, to allow popFront() (and if 
 implemented, popBack()) to return an `Option!(ElementType, 
 None)`.
 
 What is returned would be either the popped element, or None 
 if no more elements remain (with the nice by-product of no 
 longer getting assert failures if pop{Front,Back} are called 
 when the range is empty).
I think the discussion about tail() had to do with immutable ranges. Having popXxx() return something would still have it mutate the range.
Ah, gotcha, thanks, I'd missed that context. Nevertheless, I'd like the option-returning popFront to be considered on its own merits, because it seems to me to add useful distinctions between what input and forward ranges to do. On immutability: it feels like this relates at least in part to not necessarily distinguishing adequately between ranges versus containers (or more generally, ranges versus owners of memory). Have I missed anything in noting that ranges can be divided roughly along the following lines: - processing external input, where the internal state is just some small memory buffer or other collection of variables used to store the latest data read - generative mechanisms where the range elements are produced by a sequential algorithm - iteration across data stored in memory ... where immutability only really makes sense in the first place w.r.t. the last of these? And there, it doesn't really make sense that the range be const or immutable -- only that the underlying data is?
Jun 26
parent reply Paul Backus <snarwin gmail.com> writes:
On Friday, 26 June 2020 at 14:46:24 UTC, Joseph Rushton Wakeling 
wrote:
 Have I missed anything in noting that ranges can be divided 
 roughly along the following lines:

   - processing external input, where the internal state is just 
 some small
     memory buffer or other collection of variables used to 
 store the latest
     data read

   - generative mechanisms where the range elements are produced 
 by a sequential
     algorithm

   - iteration across data stored in memory

 ... where immutability only really makes sense in the first 
 place w.r.t. the last of these?  And there, it doesn't really 
 make sense that the range be const or immutable -- only that 
 the underlying data is?
Because const and immutable are transitive, any data structure that includes a range, either directly or by reference, is itself forced to be mutable. This makes D's const and immutable significantly less useful than they otherwise would be; i.e., it makes D a worse language. The fact that you, personally, cannot think of a use-case for immutable ranges off the top of your head is not a strong argument against supporting them. All language and library features ought to work together unless there is a specific reason for them not to (as is the case, for example, with garbage collection and BetterC). So I think the burden of proof is on you here. If we're designing a new range interface, is there a good reason it *shouldn't* work with const and immutable? One argument in favor of requiring mutability is that using tail() instead of popFront() could result in a loss of performance. I am working on investigating that claim using the examples provided by Jon Degenhardt earlier in this thread. Another, which I find less convincing, is that writing algorithms recursively instead of iteratively would be too "difficult" or "onerous". If you have any more to add, I am interested in hearing them. [1] https://issues.dlang.org/show_bug.cgi?id=5377
Jun 26
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Friday, 26 June 2020 at 16:03:23 UTC, Paul Backus wrote:
 The fact that you, personally, cannot think of a use-case for 
 immutable ranges off the top of your head is not a strong 
 argument against supporting them. All language and library 
 features ought to work together unless there is a specific 
 reason for them not to (as is the case, for example, with 
 garbage collection and BetterC).

 So I think the burden of proof is on you here. If we're 
 designing a new range interface, is there a good reason it 
 *shouldn't* work with const and immutable?
There's no need to be combative here. I'm not trying to argue a side, but to make sure we get clarity on what it means for ranges to be compatible with const and immutable in this way. Whether we like it or not, pretty much all ranges currently work via some under-the-hood mutability, whether it's just updating the pointer of a slice, incrementing an internal variable, or reading from some external input into an internal buffer. In some cases there are ready ways out of that. For a forward range whose internal state is cheap to duplicate, there should be no problem just (automatically) making a mutable deep copy to iterate over. If we're talking about a range that iterates over the contents of some memory locations, it seems natural enough that it should be possible to automatically create a mutable shallow copy that iterates over immutable memory locations (analogous to immutable(T[]) => immutable(T)[] and so on). But -- for example -- what would it mean to have an input range as part of a const or immutable data structure? Forget the strict interpretation of the current D keywords, let's just think conceptually for a moment. Do we want to think of an immutable input range as just a source of input, with any under-the-hood mutability just an implementation detail that shouldn't be of interest to the external caller? Do we consider it a contradiction in terms? I'm inclined to agree with the suggestion that input ranges should either be non-copyable or have reference semantics, but I don't think that really resolves the issue given the transitiveness of const and immutable. I also think it may be worth making a point of comparison to Rust's iterators. Obviously Rust has a different approach to immutability than D -- the main concern is preventing multiple mutable references to the same piece of data -- but there is a very clear separation between iterators versus what they iterate over, and generally speaking (probably because of that separation) iterators themselves seem to be always implemented as mutable.
 One argument in favor of requiring mutability is that using 
 tail() instead of popFront() could result in a loss of 
 performance. I am working on investigating that claim using the 
 examples provided by Jon Degenhardt earlier in this thread.
FWIW my objection to `tail` was probably misplaced. What I was interested in was coming from a completely different angle to the const/immutable concern. Specifically, I'm interested in drawing a clear line between input ranges -- where it is not really a given that `front` can be pre-populated -- versus forward ranges. For example, given a RandomSample, one arguably does not want `front` to be pre-determined on construction, but from the moment one starts consuming the range. Given the current range API that means some unpleasant is-this-the-first-call special-casing under the hood of the `front` method. So, what I'm interested in is whether we can reliably rework the input range API such that the only way to determine the next element is to actually fetch it. That would also work with a `tail`-like approach if the function concerned would return something like: Tuple!(Option!(ElementType, None), Tail) ... but I'm not sure if that would interfere with other concerns about that approach (e.g. would it get in the way of optimizing performance of the recursive approach?).
 Another, which I find less convincing, is that writing 
 algorithms recursively instead of iteratively would be too 
 "difficult" or "onerous". If you have any more to add, I am 
 interested in hearing them.
I don't have specific examples, but it's worth noting that one of the plus points of D is that it tends to allow you to write any given code in the simplest and most natural way, rather than forcing you through more complicated paradigms for the sake of it. So, I'd personally be reluctant to force all writers of ranges to write their code in a more complicated/virtuous way unless their use case specifically requires it. With that in mind, it might be better to identify ways to simplify the _implementation_ of ranges that need to support const/immutable use cases. That is, keep it pay-as-you-go -- you don't have to write ranges in a way that supports const/immutable if you don't have that use-case -- but if you do, the cost you have to pay is less.
Jun 27
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 27 June 2020 at 12:25:34 UTC, Joseph Rushton 
Wakeling wrote:

 But -- for example -- what would it mean to have an input range 
 as part of a const or immutable data structure?  Forget the 
 strict interpretation of the current D keywords, let's just 
 think conceptually for a moment.  Do we want to think of an 
 immutable input range as just a source of input, with any 
 under-the-hood mutability just an implementation detail that 
 shouldn't be of interest to the external caller?  Do we 
 consider it a contradiction in terms?
An immutable or const input range just cannot be. An input range can't not be mutable. If a range itself can implement a `tail()`, it's not an input range, it's a forward range. The only way for `tail()` to work with input ranges is to be a free function: R tail(R)(R r) if (isInputRangeWithAHypotheticalAPI!R) { r.next(); // fetchNext(), advance(), whatever the API is return move(r); } and used like this: input = move(input).tail; (Yes, I'm still insisting that input ranges must be non-copyable).
 I also think it may be worth making a point of comparison to 
 Rust's iterators.  Obviously Rust has a different approach to 
 immutability than D -- the main concern is preventing multiple 
 mutable references to the same piece of data -- but there is a 
 very clear separation between iterators versus what they 
 iterate over, and generally speaking (probably because of that 
 separation) iterators themselves seem to be always implemented 
 as mutable.
True input ranges are basically Rust's iterators, i.e. their API should be Optional!T next(); // in other words, a bool and a T or variations thereof, as outlined by Andrei some pages back. IMHO, the above API is the cleanest. But there is also place for buffered input ranges (i.e. the currently existing API), see below.
 So, what I'm interested in is whether we can reliably rework 
 the input range API such that the only way to determine the 
 next element is to actually fetch it.
Both APIs have their place. A "true" input range (i.e. a generator) yields temporaries, and only needs a "fetch next" primitive. But the *algorithms* may require a buffer. Consider a `find`: it needs to return the needle and the remainder of the range. In order to find the needle it needs to pop it off from the "fetch next" haystack, irreversibly advancing it. If we want to keep `find` generic over input and forward ranges (i.e. returning a range), it will have to wrap the haystack into a buffered input range with the front/popFront API, placing the found needle at the front, i.e. hypothetically: auto find(alias pred, Range, Needle)(Range range, auto ref const Needle needle) if (isInputRangeWithAHypotheticalAPI!R) { static struct Result { private typeof(Range.init.next()) buf; private Range src; // So long as the Optional holds a valid value, range is not empty property bool empty() const { return !buf; } // Accessing an empty Optional would be illegal, // so the non-empty() invariant is observed. ref front() inout { return buf.value; } void popFront() { buf = src.next(); } } typeof(range.next()) current; // i.e. an Optional!ElementType while (true) { current = range.next; if (!current || pred(current.value, needle)) break; } return Result(move(current), move(range)); }
Jun 27
parent reply Paul Backus <snarwin gmail.com> writes:
On Saturday, 27 June 2020 at 13:11:09 UTC, Stanislav Blinov wrote:
 On Saturday, 27 June 2020 at 12:25:34 UTC, Joseph Rushton 
 Wakeling wrote:

 But -- for example -- what would it mean to have an input 
 range as part of a const or immutable data structure?  Forget 
 the strict interpretation of the current D keywords, let's 
 just think conceptually for a moment.  Do we want to think of 
 an immutable input range as just a source of input, with any 
 under-the-hood mutability just an implementation detail that 
 shouldn't be of interest to the external caller?  Do we 
 consider it a contradiction in terms?
An immutable or const input range just cannot be. An input range can't not be mutable. If a range itself can implement a `tail()`, it's not an input range, it's a forward range.
This is false. While it is true that a pure input range cannot exist without the presence of mutable state *somewhere*, it does not follow that the mutable state must be stored in (or pointed to by) the range itself. Consider the following example: auto fdByChar(int fd) { import core.sys.posix.unistd; static struct Result { private int fd; bool empty; char front; Result tail() const in (!empty) { char next; ssize_t nread = read(fd, cast(void*) &next, 1); if (nread == 0) return Result(fd, true, char.init); else if (nread == 1) return Result(fd, false, next); else throw new Exception("Error reading file"); } } return Result(fd, false, char.init).tail; } This is a pure input range. Copying it and iterating the copy will invalidate the original. Yet it is perfectly capable of being `immutable`. Here's an example of usage: void each(alias fun, Range)(Range range) { if (range.empty) return; fun(range.front); range.tail.each!fun; } void main() { import std.stdio; immutable stdinByChar = fdByChar(0); stdinByChar.each!writeln; } Of course, there is mutable state involved here, but that state is not in the range--it's in the kernel. So the `immutable` qualifier on the range does not apply to it.
Jun 27
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 27 June 2020 at 17:16:10 UTC, Paul Backus wrote:

 An immutable or const input range just cannot be. An input 
 range can't not be mutable.
 If a range itself can implement a `tail()`, it's not an input 
 range, it's a forward range.
This is false. While it is true that a pure input range cannot exist without the presence of mutable state *somewhere*, it does not follow that the mutable state must be stored in (or pointed to by) the range itself. Consider the following example: auto fdByChar(int fd) ... Of course, there is mutable state involved here, but that state is not in the range--it's in the kernel. So the `immutable` qualifier on the range does not apply to it.
Thereby result of (input.tail == input.tail) is a coin flip? I don't think that's a sensible design.
Jun 27
parent reply Paul Backus <snarwin gmail.com> writes:
On Saturday, 27 June 2020 at 17:30:54 UTC, Stanislav Blinov wrote:
 Thereby result of (input.tail == input.tail) is a coin flip? I 
 don't think that's a sensible design.
Equality is not part of the range interface (current or proposed), so there is no guarantee that comparing two ranges with `==` will give you a useful result.
Jun 27
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Saturday, 27 June 2020 at 18:55:39 UTC, Paul Backus wrote:
 On Saturday, 27 June 2020 at 17:30:54 UTC, Stanislav Blinov 
 wrote:
 Thereby result of (input.tail == input.tail) is a coin flip? I 
 don't think that's a sensible design.
Equality is not part of the range interface (current or proposed), so there is no guarantee that comparing two ranges with `==` will give you a useful result.
Equality is very much part of the struct interface though. I'll restate. What about input.tail.empty == input.tail.empty? Or input.tail.front == input.tail.front? Or, speaking of ranges, a input.tail.equal(input.tail)?.. I thought you were concerned about having to explain something to beginners ;) If the explanation is "`tail() const` is impure, and calling it repeatedly with the same `this` produces different results, even when `this` is immutable", "WTF???" would be quite a reasonable follow-up question. This is not malloc. If we do define a `tail` primitive, it better yield the same output for the same input, meaning it would only be a primitive of forward ranges.
Jun 27
next sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Saturday, 27 June 2020 at 20:14:36 UTC, Stanislav Blinov wrote:
 Equality is very much part of the struct interface though. I'll 
 restate. What about input.tail.empty == input.tail.empty? Or 
 input.tail.front == input.tail.front? Or, speaking of ranges, a 
 input.tail.equal(input.tail)?..
Does it even make _sense_ to be able to call `tail` twice on the same input range instance?
 If the explanation is "`tail() const` is impure, and calling it 
 repeatedly with the same `this` produces different results,
 even when `this` is immutable", "WTF???" would be quite a 
 reasonable follow-up question.
One of the problems with thinking about ranges -- and the current range API reflects this -- is that it tends to start by thinking about deterministic examples (e.g. arrays). But while some input ranges may be deterministic in practice, it is not possible to reasonably assume "same input, same output" behaviour. That's the whole _point_ of input ranges. As an example: reading a stream of values from a crypto RNG, or a "true random" source measured from some ongoing physical process. A key thing to observe there is that, immutable or no, the state of `this` doesn't give you complete information about what comes next in the sequence, and indeed the result may also depend on precisely _when_ the next element is requested (e.g. if my input range is a "true random" noise source, I'm likely to get a different value if I wait 0.1 sec to call it rather than calling it right now, even if no one else has been using that noise source). Conceptually that doesn't fit very well with the concept of `tail`, but that's just a matter of a better name. The real problem here isn't the indeterministic results, it's the idea that the original `this` is usable after `tail` has been called on it that is the problem.
Jun 28
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Sunday, 28 June 2020 at 10:42:14 UTC, Joseph Rushton Wakeling 
wrote:
 On Saturday, 27 June 2020 at 20:14:36 UTC, Stanislav Blinov 
 wrote:
 Equality is very much part of the struct interface though. 
 I'll restate. What about input.tail.empty == input.tail.empty? 
 Or input.tail.front == input.tail.front? Or, speaking of 
 ranges, a input.tail.equal(input.tail)?..
Does it even make _sense_ to be able to call `tail` twice on the same input range instance?
No, it does not, that's my point. "Don't do that" isn't the answer. "Can't do that" (statically disallowed, not present in the interface) is the answer.
 If the explanation is "`tail() const` is impure, and calling 
 it repeatedly with the same `this` produces different results,
 even when `this` is immutable", "WTF???" would be quite a 
 reasonable follow-up question.
One of the problems with thinking about ranges -- and the current range API reflects this -- is that it tends to start by thinking about deterministic examples (e.g. arrays).
Arrays are random access ranges (which are also forward). They're not input ranges :) The current range API incorrectly treats any range as at least an input range. If we are talking about making changes to range API - that is one of the changes that needs to be made.
 But while some input ranges may be deterministic in practice, 
 it is not possible to reasonably assume "same input, same 
 output" behaviour.  That's the whole _point_ of input ranges.
Yes, that is why an input range cannot be implementing `tail`. You cannot keep the original range *and* the tail. That's what forward ranges do.
 As an example: reading a stream of values from a crypto RNG, or 
 a "true random" source measured from some ongoing physical 
 process.

 A key thing to observe there is that, immutable or no, the 
 state of `this` doesn't give you complete information about 
 what comes next in the sequence, and indeed the result may also 
 depend on precisely _when_ the next element is requested (e.g. 
 if my input range is a "true random" noise source, I'm likely 
 to get a different value if I wait 0.1 sec to call it rather 
 than calling it right now, even if no one else has been using 
 that noise source).
 Conceptually that doesn't fit very well with the concept of 
 `tail`, but that's just a matter of a better name.
It isn't. It's a matter of semantics of the range. You can only consume or discard an input range. You can't fork it. Conversely, if you *can* fork a range - it's a forward range. Therefore, an input range cannot have a `tail` :)
 The real problem here isn't the indeterministic results, it's 
 the idea that the original `this` is usable after `tail` has 
 been called on it that is the problem.
We cannot, in the language, statically, express a "usability" of an lvalue.
Jun 28
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Sunday, 28 June 2020 at 11:33:39 UTC, Stanislav Blinov wrote:
 Does it even make _sense_ to be able to call `tail` twice on 
 the same input range instance?
No, it does not, that's my point. "Don't do that" isn't the answer. "Can't do that" (statically disallowed, not present in the interface) is the answer.
Indeed. You seem to be thinking I'm disagreeing with you, when actually we are in agreement on this point ;-)
 One of the problems with thinking about ranges -- and the 
 current range API reflects this -- is that it tends to start 
 by thinking about deterministic examples (e.g. arrays).
Arrays are random access ranges (which are also forward). They're not input ranges :)
That's not strictly true: one can reasonably _interpret_ an array as an input range if that's what is desired. What makes an input range distinct is not whether the actual elements come from some impure input source (it can be deterministic under the hood, whether from iterating over array elements, or a deterministic generative mechanism like a PRNG, or whatever), but what assumptions the caller can make about how to use and interact with it.
 The current range API incorrectly treats any range as at least 
 an input range. If we are talking about making changes to range 
 API - that is one of the changes that needs to be made.
That doesn't really make sense to me. Why is it not possible to interact with any other range via input range semantics?
 But while some input ranges may be deterministic in practice, 
 it is not possible to reasonably assume "same input, same 
 output" behaviour.  That's the whole _point_ of input ranges.
Yes, that is why an input range cannot be implementing `tail`. You cannot keep the original range *and* the tail. That's what forward ranges do.
Right, _you cannot keep the original range and the tail_. But that doesn't block you having a method which takes the original range as input, and returns the updated range. The issue is stopping you using the _original_ range (e.g. requiring it to be passed into the method via a move rather than by reference?).
Jun 28
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Sunday, 28 June 2020 at 12:43:53 UTC, Joseph Rushton Wakeling 
wrote:
 On Sunday, 28 June 2020 at 11:33:39 UTC, Stanislav Blinov wrote:
 Does it even make _sense_ to be able to call `tail` twice on 
 the same input range instance?
No, it does not, that's my point. "Don't do that" isn't the answer. "Can't do that" (statically disallowed, not present in the interface) is the answer.
Indeed. You seem to be thinking I'm disagreeing with you, when actually we are in agreement on this point ;-)
Ah, that wasn't evident from the question :*)
 One of the problems with thinking about ranges -- and the 
 current range API reflects this -- is that it tends to start 
 by thinking about deterministic examples (e.g. arrays).
Arrays are random access ranges (which are also forward). They're not input ranges :)
That's not strictly true: one can reasonably _interpret_ an array as an input range if that's what is desired.
That does not make arrays input ranges. They remain random-access ranges.
 What makes an input range distinct is not whether the actual 
 elements come from some impure input source (it can be 
 deterministic under the hood, whether from iterating over array 
 elements, or a deterministic generative mechanism like a PRNG, 
 or whatever), but what assumptions the caller can make about 
 how to use and interact with it.
That would be design by convention, which I'm strongly opposed to, and arguing against. There's no need to *document* caller's assumptions when we can devise an API that drives them. What makes an input range distinct is that, unlike other kinds, it can only be consumed once - there's no need for any other assumptions. The current range API does not enforce this in any way, hence the copying conundrum and Jonathan's "don't use original after you make a copy" or other similar statements that amount to "you should know what you're doing".
 The current range API incorrectly treats any range as at least 
 an input range. If we are talking about making changes to 
 range API - that is one of the changes that needs to be made.
That doesn't really make sense to me. Why is it not possible to interact with any other range via input range semantics?
Because they have different semantics! The current range API attempts to differentiate solely on interface (which can indeed appear intersecting) but not semantics. What I meant is, today in Phobos, there are algorithms that do this: auto algo(Range)(Range range) if (isInputRange!Range) { /* ... */ static if (isForwardRange!Range) { /* ... */ } } Input ranges are treated as a subset of forward ranges. But they're not - they're distinct, non-intersecting sets. If we do change the API, the above will have to become either (depending on the algorithm): auto algo(Range)(Range range) if (isInputRange!Range) { /* ... */ } auto algo(Range)(Range range) if (isForwardRange!Range) { /* ... */ } // OR, given a hypothetical `isSomeRange` test auto algo(Range)(Range range) if (isSomeRange!Range) // note: isSomeRange, not isInputRange { /* use common API, if any ... */ static if (isForwardRange!Range) { /* ... */ } }
 Right, _you cannot keep the original range and the tail_.  But 
 that doesn't block you having a method which takes the original 
 range as input, and returns the updated range.  The issue is 
 stopping you using the _original_ range (e.g. requiring it to 
 be passed into the method via a move rather than by reference?).
Yes, i.e. the free function implementation of `tail` that I posted previously (which would work correctly for both input ranges and forward ranges, *assuming* input ranges aren't copyable and forward ranges are).
Jun 28
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sun, Jun 28, 2020 at 02:00:42PM +0000, Stanislav Blinov via Digitalmars-d
wrote:
[...]
 What makes an input range distinct is that, unlike other kinds, it can
 only be consumed once - there's no need for any other assumptions. The
 current range API does not enforce this in any way, hence the copying
 conundrum and Jonathan's "don't use original after you make a copy" or
 other similar statements that amount to "you should know what you're
 doing".
This is not a sufficient assumption, because there remains the question of what state a range should be in after you iterate over it partially, say, using a foreach loop: MyRange r = ...; foreach (e; r) { if (someCondition) break; } foreach (e; r) { // Do we get the remaining elements of r here, or do we // get all the elements from the beginning again? } Even if we grant that r is a forward range, the semantics of the above code is still underspecified. And it's not just a matter of specifying that the semantics should be; there's also the question of whether we should specify anything at all, keeping in mind that the more specific the definition, the less room there is for efficient implementation of ranges (i.e., some optimizations may be excluded if there are too many semantic requirements on ranges). [...]
 Input ranges are treated as a subset of forward ranges. But they're
 not - they're distinct, non-intersecting sets.
OTOH, there is a great deal of advantage in using the subset of semantics in which input ranges *are* subsets of forward ranges, because this allows us to unify a lot of iteration logic. Otherwise, like you say below, every algorithm will need to be duplicated, once for input ranges, once for forward ranges, which will instantly double the size of Phobos algorithms. [...]
 auto algo(Range)(Range range)
 if (isInputRange!Range)
 { /* ... */ }
 
 auto algo(Range)(Range range)
 if (isForwardRange!Range)
 { /* ... */ }
[...] T -- If it tastes good, it's probably bad for you.
Jun 28
parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Sunday, 28 June 2020 at 15:07:21 UTC, H. S. Teoh wrote:
 On Sun, Jun 28, 2020 at 02:00:42PM +0000, Stanislav Blinov via 
 Digitalmars-d wrote: [...]
 What makes an input range distinct is that, unlike other 
 kinds, it can only be consumed once - there's no need for any 
 other assumptions. The current range API does not enforce this 
 in any way, hence the copying conundrum and Jonathan's "don't 
 use original after you make a copy" or other similar 
 statements that amount to "you should know what you're doing".
This is not a sufficient assumption, because there remains the question of what state a range should be in after you iterate over it partially, say, using a foreach loop: MyRange r = ...; foreach (e; r) { if (someCondition) break; } foreach (e; r) { // Do we get the remaining elements of r here, or do we // get all the elements from the beginning again? }
 Even if we grant that r is a forward range, the semantics of 
 the above code is still underspecified.
Of course it isn't, because there's a problem with `foreach`. It iterates over a copy, except if there's an opApply. I'm not sure what your argument is. I repeat: "The current range API does not enforce [input range behavior] in any way, hence the copying conundrum..." `foreach` is very much a part of range API.
 And it's not just a matter of specifying that the semantics 
 should be; there's also the question of whether we should 
 specify anything at all, keeping in mind that the more specific 
 the definition, the less room there is for efficient 
 implementation of ranges (i.e., some optimizations may be 
 excluded if there are too many semantic requirements on ranges).
Yes, we should. For example, non-copyability of input ranges, which enables *more* efficient implementations (whereas expecting them to remain copyable dictates the opposite).
 ...there is a great deal of advantage in using the subset of 
 semantics in which input ranges *are* subsets of forward 
 ranges, because this allows us to unify a lot of iteration 
 logic.
That's not based on range semantics, it's based solely on interface. An input range and a forward range can have the same interface for iteration, i.e. a `fetchNext` or its variant. The only difference is that you can't fork the former.
 Otherwise, like you say below, every algorithm will need to be 
 duplicated, once for input ranges, once for forward ranges, 
 which will instantly double the size of Phobos algorithms.
That's a gross exaggerration. Algorithms already do the semantic duplication, for arrays, for hasLvalueElements, etc., etc., ultimately, when everything else fails, arriving at the "while (!r.empty) r.popFront()".
Jun 28
prev sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Saturday, 27 June 2020 at 20:14:36 UTC, Stanislav Blinov wrote:
 If the explanation is "`tail() const` is impure, and calling it 
 repeatedly with the same `this` produces different results, 
 even when `this` is immutable", "WTF???" would be quite a 
 reasonable follow-up question. This is not malloc. If we do 
 define a `tail` primitive, it better yield the same output for 
 the same input, meaning it would only be a primitive of forward 
 ranges.
It sounds like what you are really trying to say here is that input ranges and forward ranges should not use the same interface, and that input ranges should instead implement only `Option!T next()`. I agree that this would be ideal, and that if this change were made, tail() would be required only for forward ranges and up. In that case, requiring tail() to be pure would make sense. My example was a response to the specific claim that "an immutable input range cannot exist." Specifically, it was a counterexample demonstrating that the claim is false. As I'm sure you can see, the claim is still false even if we agree to use next() instead of tail() for input ranges, since next() is obviously allowed to be impure.
Jun 28
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Sunday, 28 June 2020 at 14:12:39 UTC, Paul Backus wrote:

 It sounds like what you are really trying to say here is that 
 input ranges and forward ranges should not use the same 
 interface, and that input ranges should instead implement only 
 `Option!T next()`.
Not exactly. A given forward range may also implement such a `next`. Taking your example, with a seekable file and `pread`. Which would, of course, narrow the set of "files" it would be able to work with. A forward range need not, necessarily, have a `front` and a `popFront` (see that funky List that Andrei posted a while back). We should differentiate by some other artifact. I maintain that artifact should be copyability. Or we can keep the distinction based on the presence of save(), and live forever with the ambiguous semantics and the "don't use original after copy unless you know what you're doing" convention (yuck). Are there even other options besides accepting UB?
 My example was a response to the specific claim that "an 
 immutable input range cannot exist." Specifically, it was a 
 counterexample demonstrating that the claim is false. As I'm 
 sure you can see, the claim is still false even if we agree to 
 use next() instead of tail() for input ranges, since next() is 
 obviously allowed to be impure.
I can't see that. I don't see how you're falsifying that claim when in order to implement the range you need mutable state. "It's mutable, but the compiler can't see that, therefore it's immutable" - that is a dangerous justification. There are indeed some legitimate applications for it (allocators, mutexes), but this isn't one of them. If `a.tail` is not returning the tail of `a`, then it's either a bug in the design (shouldn't be called `tail`) or in the implementation (should not mutate *any* state).
Jun 28
parent reply Paul Backus <snarwin gmail.com> writes:
On Sunday, 28 June 2020 at 15:33:48 UTC, Stanislav Blinov wrote:
 On Sunday, 28 June 2020 at 14:12:39 UTC, Paul Backus wrote:

 It sounds like what you are really trying to say here is that 
 input ranges and forward ranges should not use the same 
 interface, and that input ranges should instead implement only 
 `Option!T next()`.
Not exactly. A given forward range may also implement such a `next`. [...] We should differentiate by some other artifact. I maintain that artifact should be copyability. Or we can keep the distinction based on the presence of save(), and live forever with the ambiguous semantics and the "don't use original after copy unless you know what you're doing" convention (yuck). Are there even other options besides accepting UB?
If input ranges are only required to implement next(), then the distinction will be that forward ranges implement head()/tail() and input ranges don't. (In this scenario, head() and tail() will of course be required to return the same result if called multiple times.) For *some* forward ranges, it will be possible to implement next() as well, in which case those ranges will be capable of functioning as both input ranges and forward ranges. I don't see anything wrong with that.
 My example was a response to the specific claim that "an 
 immutable input range cannot exist." Specifically, it was a 
 counterexample demonstrating that the claim is false. As I'm 
 sure you can see, the claim is still false even if we agree to 
 use next() instead of tail() for input ranges, since next() is 
 obviously allowed to be impure.
I can't see that. I don't see how you're falsifying that claim when in order to implement the range you need mutable state.
Ok, I'll show you. Here's the same range implemented using next() instead of front() and tail(): // Just enough for the example to compile struct Optional(T) { bool empty; T value; } Optional!T some(T)(T value) { return Optional!T(false, value); } Optional!T no(T)() { return Optional!T(true, T.init); } auto fdByChar(int fd) { import core.sys.posix.unistd; static struct Result { private int fd; Optional!char next() const { char c; ssize_t nread = read(fd, cast(void*) &c, 1); if (nread == 0) return no!char; else if (nread == 1) return some(c); else throw new Exception("Error reading file"); } } return Result(fd); } And here's the usage: void each(alias fun, Range)(Range range) { for (auto e = range.next; !e.empty; e = range.next) fun(e.value); } void main() { import std.stdio; immutable stdinByChar = fdByChar(0); stdinByChar.each!writeln; } Once again, `stdinByChar` is an input range, and it is immutable. QED.
Jun 28
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Sunday, 28 June 2020 at 16:37:23 UTC, Paul Backus wrote:
 On Sunday, 28 June 2020 at 15:33:48 UTC, Stanislav Blinov wrote:
 On Sunday, 28 June 2020 at 14:12:39 UTC, Paul Backus wrote:

 It sounds like what you are really trying to say here is that 
 input ranges and forward ranges should not use the same 
 interface, and that input ranges should instead implement 
 only `Option!T next()`.
Not exactly. A given forward range may also implement such a `next`. [...] We should differentiate by some other artifact. I maintain that artifact should be copyability. Or we can keep the distinction based on the presence of save(), and live forever with the ambiguous semantics and the "don't use original after copy unless you know what you're doing" convention (yuck). Are there even other options besides accepting UB?
If input ranges are only required to implement next(), then the distinction will be that forward ranges implement head()/tail() and input ranges don't. (In this scenario, head() and tail() will of course be required to return the same result if called multiple times.)
Just as forward ranges may implement `next()`, some input ranges may want to be buffered, i.e.: struct Wrapped { private typeof(src.next()) buf; // Optional!ElementType private Input src; // Input is a `next()` input range property bool empty() const { return !buf; } T moveFront() { return buf.replace(T.init); } void popFront() { buf = src.next; } // no front, only moveFront } If we put a hard requirement on input ranges to *only* exist with the next() interface, this will shut down possible optimizations that buffering enables. We can't implement a buffering range in terms of the `next()` interface, a staggered fetch + advance is required. (next() may overwrite the internals of the buffer before client has a chance of reading the value).
 My example was a response to the specific claim that "an 
 immutable input range cannot exist." Specifically, it was a 
 counterexample demonstrating that the claim is false. As I'm 
 sure you can see, the claim is still false even if we agree 
 to use next() instead of tail() for input ranges, since 
 next() is obviously allowed to be impure.
I can't see that. I don't see how you're falsifying that claim when in order to implement the range you need mutable state.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 Ok, I'll show you. Here's the same range implemented using 
 next() instead of front() and tail():

             Optional!char next() const
             {
                 char c;
                 ssize_t nread = read(fd, cast(void*) &c, 1);
^^^^
                 if (nread == 0)
                     return no!char;
                 else if (nread == 1)
                     return some(c);
                 else
                     throw new Exception("Error reading file");
             }
         }
 Once again, `stdinByChar` is an input range, and it is 
 immutable. QED.
:) Fine. An input range can't exist without mutable state. Is that statement more agreeable for you?
Jun 29
parent reply Paul Backus <snarwin gmail.com> writes:
On Monday, 29 June 2020 at 15:40:28 UTC, Stanislav Blinov wrote:
 We can't implement a buffering range in terms of the `next()` 
 interface, a staggered fetch + advance is required. (next() may 
 overwrite the internals of the buffer before client has a 
 chance of reading the value).
Are you referring to ranges like `byLine` that invalidate the previous front when popFront is called? If so, I don't see what difference it makes whether they're implemented with next() or some other interface. If that's not what you're referring to, can you give an example?
 :) Fine. An input range can't exist without mutable state. Is 
 that statement more agreeable for you?
Yes, that's much more agreeable. :) Personally, I am concerned here with how to make ranges work with D's `const` and `immutable` qualifiers. So the fact that there may be mutable state out there somewhere in libc/the kernel/the filesystem doesn't bother me.
Jun 29
parent Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Monday, 29 June 2020 at 17:49:29 UTC, Paul Backus wrote:

 Are you referring to ranges like `byLine` that invalidate the 
 previous front when popFront is called?
Yes.
 If so, I don't see what difference it makes whether they're 
 implemented with next() or some other interface.
Efficiency. A given algorithm may need to stagger the range in order to preserve the head. To keep a `next()` interface and avoid buffer corruption, additional branch is needed for staggering (see [1]). This can be avoided by switching to front/popFront buffering where caller can explicitly announce they're done reading the element (not in the gist (yet)). [1] https://gist.github.com/radcapricorn/d76d29c6df6fa822d7889e799937f39d#file-inputrange-d-L180
 Personally, I am concerned here with how to make ranges work 
 with D's `const` and `immutable` qualifiers. So the fact that 
 there may be mutable state out there somewhere in libc/the 
 kernel/the filesystem doesn't bother me.
I understand that. Forward ranges are good candidates for const/immutable. Input ranges though - I'd prefer to tread lightly there.
Jun 29
prev sibling parent reply Mathias LANG <geod24 gmail.com> writes:
On Monday, 22 June 2020 at 21:46:51 UTC, Andrei Alexandrescu 
wrote:
 On 6/22/20 12:50 PM, Paul Backus wrote:

 IMHO the principled way to allow user-defined implicit 
 conversions is...to allow user-defined implicit conversions. 
 But iirc that's a can of worms Walter prefers not to open.
What happens upon function calls is not an implicit conversion. It's a forced type change.
Can you expand on this ? I've never heard of this before.
Jun 22
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/22/20 11:22 PM, Mathias LANG wrote:
 On Monday, 22 June 2020 at 21:46:51 UTC, Andrei Alexandrescu wrote:
 On 6/22/20 12:50 PM, Paul Backus wrote:

 IMHO the principled way to allow user-defined implicit conversions 
 is...to allow user-defined implicit conversions. But iirc that's a 
 can of worms Walter prefers not to open.
What happens upon function calls is not an implicit conversion. It's a forced type change.
Can you expand on this ? I've never heard of this before.
https://run.dlang.io/is/KgeosK Yah it's kinda surprising innit. For regular functions it's business as usual - exact match is attempted, conversions are considered etc. const(T[]) is convertible to const(T)[], nothing new about that. For template functions however, although templates always do exact match, arrays surreptitiously change their type from const(T[]) to const(T)[], without a conversion in sight. We need to offer that chance to other ranges if we want them to enjoy a similar treatment.
Jun 22
parent reply Mathias LANG <geod24 gmail.com> writes:
On Tuesday, 23 June 2020 at 05:26:21 UTC, Andrei Alexandrescu 
wrote:
 On 6/22/20 11:22 PM, Mathias LANG wrote:
 On Monday, 22 June 2020 at 21:46:51 UTC, Andrei Alexandrescu 
 wrote:
 On 6/22/20 12:50 PM, Paul Backus wrote:

 IMHO the principled way to allow user-defined implicit 
 conversions is...to allow user-defined implicit conversions. 
 But iirc that's a can of worms Walter prefers not to open.
What happens upon function calls is not an implicit conversion. It's a forced type change.
Can you expand on this ? I've never heard of this before.
https://run.dlang.io/is/KgeosK Yah it's kinda surprising innit. For regular functions it's business as usual - exact match is attempted, conversions are considered etc. const(T[]) is convertible to const(T)[], nothing new about that. For template functions however, although templates always do exact match, arrays surreptitiously change their type from const(T[]) to const(T)[], without a conversion in sight. We need to offer that chance to other ranges if we want them to enjoy a similar treatment.
So it's not on function call, but it's about the type being deduced. There's quite a big difference here. To me this makes perfect sense: This little peculiarity allows us to reduce the amount of template instantiations. E.g. the following code will only instantiate a single template: ``` void fun2(T)(const T[] a) { pragma(msg, T.stringof); } void main() { immutable(int[]) a; fun2(a); int[] b; fun2(b); } ``` In fact I'd love if we extend that to `auto` so the following code would work: ``` immutable(int[]) fun2() { return null; } void main() { auto x = fun2(); x = fun2(); // Currently `x` is `immutable(int[])` not `immutable(int)[]` } ```
Jun 23
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/23/20 5:45 AM, Mathias LANG wrote:
 On Tuesday, 23 June 2020 at 05:26:21 UTC, Andrei Alexandrescu wrote:
 https://run.dlang.io/is/KgeosK

 Yah it's kinda surprising innit. For regular functions it's business 
 as usual - exact match is attempted, conversions are considered etc. 
 const(T[]) is convertible to const(T)[], nothing new about that.

 For template functions however, although templates always do exact 
 match, arrays surreptitiously change their type from const(T[]) to 
 const(T)[], without a conversion in sight.

 We need to offer that chance to other ranges if we want them to enjoy 
 a similar treatment.
So it's not on function call, but it's about the type being deduced. There's quite a big difference here.
Yes, I agree. It works because of the implicit conversion.
 
 To me this makes perfect sense: This little peculiarity allows us to 
 reduce the amount of template instantiations. E.g. the following code 
 will only instantiate a single template:
 ```
 void fun2(T)(const T[] a) { pragma(msg, T.stringof); }
 
 void main() {
      immutable(int[]) a;
      fun2(a);
      int[] b;
      fun2(b);
 }
 ```
No, that would happen anyway, because in that case, T is stripped of the type modifier. It's this case where it affects the result: void fun(T)(T a) { pragma(msg, T.stringof); } void main() { const int[] a; const(int)[] b; fun(a); fun(b); } You only get one instantiation there. The case makes complete sense, because the head is always a different memory location (a copy). What we lack is a way to make the head mutable of any type via a type modifier, so it's a "special case" in the sense that it's possible only on T[] and T*. If we change the special case to a general case, then it naturally makes things a lot more usable, and gets rid of a lot of the complaints about const ranges.
 
 In fact I'd love if we extend that to `auto` so the following code would 
 work:
 ```
 immutable(int[]) fun2() { return null; }
 
 void main() {
      auto x = fun2();
      x = fun2(); // Currently `x` is `immutable(int[])` not 
 `immutable(int)[]`
 }
 ```
That would be reasonable too, IMO. -Steve
Jun 23
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/23/20 8:19 AM, Steven Schveighoffer wrote:
 On 6/23/20 5:45 AM, Mathias LANG wrote:
 On Tuesday, 23 June 2020 at 05:26:21 UTC, Andrei Alexandrescu wrote:
 https://run.dlang.io/is/KgeosK

 Yah it's kinda surprising innit. For regular functions it's business 
 as usual - exact match is attempted, conversions are considered etc. 
 const(T[]) is convertible to const(T)[], nothing new about that.

 For template functions however, although templates always do exact 
 match, arrays surreptitiously change their type from const(T[]) to 
 const(T)[], without a conversion in sight.

 We need to offer that chance to other ranges if we want them to enjoy 
 a similar treatment.
So it's not on function call, but it's about the type being deduced. There's quite a big difference here.
Yes, I agree. It works because of the implicit conversion.
No, it doesn't work because of the implicit conversion, although implicit conversion is part of it. It works because there's a hack in the compiler that introduces the conversion from const(T[]) to const(T)[] automatically for template functions. It never happens for any other type - templates in D bind types "perfectly" otherwise. This is one exception.
 It's this case where it affects the result:
 
 void fun(T)(T a) { pragma(msg, T.stringof); }
 
 void main()
 {
     const int[] a;
     const(int)[] b;
     fun(a);
     fun(b);
 }
 
 You only get one instantiation there.
Yes, this is illustrating the hack.
 The case makes complete sense, because the head is always a different 
 memory location (a copy). What we lack is a way to make the head mutable 
 of any type via a type modifier, so it's a "special case" in the sense 
 that it's possible only on T[] and T*. If we change the special case to 
 a general case, then it naturally makes things a lot more usable, and 
 gets rid of a lot of the complaints about const ranges.
Correct, thanks. That's where T.opOnCall() would fit in - the function would effect a change of type when an object of type T is passed by value to a template function. It sounds awfully particular, but I'm encouraged by the fact that it has worked so well for so many years with arrays and pointers.
Jun 23
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/23/20 11:54 AM, Andrei Alexandrescu wrote:
 No, it doesn't work because of the implicit conversion, although 
 implicit conversion is part of it.
Whether the compiler hacks it by converting the type before invoking the type detection, or whether it hacks by adjusting the IFTI detection is not really important. In the general case, though, there is no implicit conversion -- the conversion function must be used. So it would be important to call the conversion function before IFTI is invoked. But another solution would be define implicit conversions that are preferred for IFTI/auto. Like an opCanonical or something. This I think sits better and is more consistent. There are other cases aside from tail-const that could make IFTI more usable. For example: https://issues.dlang.org/show_bug.cgi?id=4998 This wouldn't be doable without some more help from the compiler during IFTI with possibly template constraints. -Steve
Jun 23
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 6/23/20 1:01 PM, Steven Schveighoffer wrote:
 On 6/23/20 11:54 AM, Andrei Alexandrescu wrote:
 No, it doesn't work because of the implicit conversion, although 
 implicit conversion is part of it.
Whether the compiler hacks it by converting the type before invoking the type detection, or whether it hacks by adjusting the IFTI detection is not really important.
*nod*
 In the general case, though, there is no implicit conversion -- the 
 conversion function must be used. So it would be important to call the 
 conversion function before IFTI is invoked.
 
 But another solution would be define implicit conversions that are 
 preferred for IFTI/auto. Like an opCanonical or something. This I think 
 sits better and is more consistent.
Yah, that's what I meant with opOnCall. opOnAutoOrIFTI...
 There are other cases aside from tail-const that could make IFTI more 
 usable. For example: https://issues.dlang.org/show_bug.cgi?id=4998
 
 This wouldn't be doable without some more help from the compiler during 
 IFTI with possibly template constraints.
*nod*
Jun 23
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/23/20 5:45 AM, Mathias LANG wrote:
 On Tuesday, 23 June 2020 at 05:26:21 UTC, Andrei Alexandrescu wrote:
 On 6/22/20 11:22 PM, Mathias LANG wrote:
 On Monday, 22 June 2020 at 21:46:51 UTC, Andrei Alexandrescu wrote:
 On 6/22/20 12:50 PM, Paul Backus wrote:

 IMHO the principled way to allow user-defined implicit conversions 
 is...to allow user-defined implicit conversions. But iirc that's a 
 can of worms Walter prefers not to open.
What happens upon function calls is not an implicit conversion. It's a forced type change.
Can you expand on this ? I've never heard of this before.
https://run.dlang.io/is/KgeosK Yah it's kinda surprising innit. For regular functions it's business as usual - exact match is attempted, conversions are considered etc. const(T[]) is convertible to const(T)[], nothing new about that. For template functions however, although templates always do exact match, arrays surreptitiously change their type from const(T[]) to const(T)[], without a conversion in sight. We need to offer that chance to other ranges if we want them to enjoy a similar treatment.
So it's not on function call, but it's about the type being deduced. There's quite a big difference here.
I don't understand the difference. The change $qual(T[]) -> $qual(T)[] happens specifically when a template parameter type is deduced for a function call.
 To me this makes perfect sense: This little peculiarity allows us to 
 reduce the amount of template instantiations.
As I said before, I appreciate there's no shortage of people ready to explain things I've done back to me. In this case it's actually Walter's work. A long time ago, shortly after we introduced ranges and algorithms to Phobos, people complained (including in this forum) that there was no way to use algorithms with constant arrays. Walter and I talked about it for a while and he introduced this glorious hack, only for arrays and only for template function calls. Suddenly const arrays started to work, and nobody complained about the lack of generality of the hack. Until now I guess!
 E.g. the following code 
 will only instantiate a single template:
 ```
 void fun2(T)(const T[] a) { pragma(msg, T.stringof); }
 
 void main() {
      immutable(int[]) a;
      fun2(a);
      int[] b;
      fun2(b);
 }
 ```
This example is actually not illustrative. The first call involves a conversion from immutable(int[]) to immutable(int)[]. This happens even for non-templates. The second call involves a conversion from int[] to const(int[]), again a standard conversion. Neither call involves the hack discussed here, which is const(int[]) to const(int)[].
 In fact I'd love if we extend that to `auto` so the following code would 
 work:
 ```
 immutable(int[]) fun2() { return null; }
 
 void main() {
      auto x = fun2();
      x = fun2(); // Currently `x` is `immutable(int[])` not 
 `immutable(int)[]`
 }
 ```
That'd work but would not improve the problem of working with const ranges.
Jun 23
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/23/20 11:50 AM, Andrei Alexandrescu wrote:
 E.g. the following code will only instantiate a single template:
 ```
 void fun2(T)(const T[] a) { pragma(msg, T.stringof); }

 void main() {
      immutable(int[]) a;
      fun2(a);
      int[] b;
      fun2(b);
 }
 ```
This example is actually not illustrative. The first call involves a conversion from immutable(int[]) to immutable(int)[].
Oops, I meant from immutable(int[]) to const(int[]).
Jun 23
prev sibling next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Monday, 22 June 2020 at 14:33:53 UTC, Steven Schveighoffer 
wrote:
 [snip]

 FeepingCreature is right, we should try and create a head 
 mutable mechanism. I have envisioned it in the past as a 
 tail-modifier mechanism (e.g. tail-const).

 -Steve
I just read a thread from 2010 on tail const/head mutable. Kind of amazing that it is a difficult enough problem that it still hasn't been resolved.
Jun 22
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 6/22/20 11:07 AM, jmh530 wrote:
 On Monday, 22 June 2020 at 14:33:53 UTC, Steven Schveighoffer wrote:
 [snip]

 FeepingCreature is right, we should try and create a head mutable 
 mechanism. I have envisioned it in the past as a tail-modifier 
 mechanism (e.g. tail-const).

 -Steve
I just read a thread from 2010 on tail const/head mutable. Kind of amazing that it is a difficult enough problem that it still hasn't been resolved.
I don't think it's that difficult, it's that it has a rather large footprint. Qualifiers weigh heavily on the language. I think going the lowering route with opCall() would be a much easier route, and also more expressive.
Jun 22
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 6/22/20 10:33 AM, Steven Schveighoffer wrote:
 On 6/22/20 10:20 AM, Paul Backus wrote:
 On Monday, 22 June 2020 at 12:15:39 UTC, Steven Schveighoffer wrote:
 My question wasn't about how such a thing could be implemented, but 
 how it works with const ranges.

 foreach(x; someConstRange) I think wouldn't be possible. I think 
 you'd have to recurse:

 void process(const Range r)
 {
    subProcess(r.front);
    process(r.rest);
 }

 The point is to question the statement "so that we can have `const` 
 and `immutable` ranges".

 Sure, we could implement recursive versions of find, etc. I don't 
 know if that's worth it.
Well, currently, range algorithms can't work with const ranges *at all*, recursively or iteratively. So from a user perspective, this would be a strict improvement on the status quo.
Algorithms can work with const ranges -- as long as the range is an array: const int[] arr = [1, 2, 3, 4, 5]; auto x = arr.find(3); assert(x == [3, 4, 5]); I think the better option is to focus on making it possible to duplicate this possibility for generic ranges rather than implement a new and awkward API. FeepingCreature is right, we should try and create a head mutable mechanism. I have envisioned it in the past as a tail-modifier mechanism (e.g. tail-const).
Oh, indeed. Arrays are special cased by the compiler - const(T[]) is silently converted to const(T)[] when calling a function. At a point there was a discussion about allowing a similar conversion to be done automatically by the compiler - opCall(). So whenever you pass an object to a function, if the type defines opCall, it would be automatically invoked.
Jun 22
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Jun 22, 2020 at 11:52:47AM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:
[...]
 At a point there was a discussion about allowing a similar conversion
 to be done automatically by the compiler - opCall(). So whenever you
 pass an object to a function, if the type defines opCall, it would be
 automatically invoked.
This conflicts with the function call operator, which is also called .opCall. I wouldn't want the function objects I pass around to be "accidentally" invoked just because it defines .opCall! T -- In a world without fences, who needs Windows and Gates? -- Christian Surchi
Jun 22
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 6/22/20 12:01 PM, H. S. Teoh wrote:
 On Mon, Jun 22, 2020 at 11:52:47AM -0400, Andrei Alexandrescu via
Digitalmars-d wrote:
 [...]
 At a point there was a discussion about allowing a similar conversion
 to be done automatically by the compiler - opCall(). So whenever you
 pass an object to a function, if the type defines opCall, it would be
 automatically invoked.
This conflicts with the function call operator, which is also called .opCall. I wouldn't want the function objects I pass around to be "accidentally" invoked just because it defines .opCall!
Oh, sorry. The name was different - possibly opOnCall. All in all it's a matter of deciding on how important this problem is. (I think it is.)
Jun 22
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/20/20 12:34 AM, H. S. Teoh wrote:
 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu via
Digitalmars-d wrote:
 [...]
 One good goal for std.v2020 would be to forego autodecoding
 throughout.
[...] Another could be to fix up the range API -- i.e, reconsider the ugliness that is .save, now that D has copy ctors.
Awesome. So we have a lil list already.
Jun 20
parent Jon Degenhardt <jond noreply.com> writes:
On Saturday, 20 June 2020 at 14:44:58 UTC, Andrei Alexandrescu 
wrote:
 On 6/20/20 12:34 AM, H. S. Teoh wrote:
 On Fri, Jun 19, 2020 at 09:14:30PM -0400, Andrei Alexandrescu 
 via Digitalmars-d wrote:
 [...]
 One good goal for std.v2020 would be to forego autodecoding
 throughout.
[...] Another could be to fix up the range API -- i.e, reconsider the ugliness that is .save, now that D has copy ctors.
Awesome. So we have a lil list already.
These two are high on my list as well. Also high on my list is a hash table in the standard library that better manages memory for large numbers of entries. There are multi-second GC pauses when large numbers of entries are added to the current AAs. There was an attempt at addressing this a couple years ago, but the first try didn't pan out: https://github.com/dlang/druntime/pull/1929. Might not require a new library version to address this.
Jun 21
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, June 22, 2020 9:37:51 PM MDT Jonathan M Davis via Digitalmars-d 
wrote:
 Now, as far as basic input ranges go, having them be classes isn't
 necessarily a problem, since basic input ranges are essentially reference
 types by their very nature (or at least, they can't be value types). If they
 could have value semantics, they could be forward ranges. In fact, in some
 ways, it would be nice to require that basic input ranges be classes, since
 then it would be easy to introspect a basic input range vs a forward range,
 and it would ensure that basic input ranges have full-on reference
 semantics, but it also would result in heap allocations that are not
 currently necessary - particularly in cases where it would currently work
 to use a pseudo-reference type rather than full-on reference type. So,
 while it would be very nice to be able to require that basic input ranges
 be classes, I doubt that it's actually reasonable to do so. Either way, it
 shouldn't be a problem to allow basic input ranges to be classes. It's just
 with forward ranges that it's a problem, but struct wrappers should solve
 that problem.
On further reflection though, one advantage of disallowing classes as ranges entirely (thus requiring a struct wrapper even for basic input ranges) would be that we could require that the init value be a valid, empty range, which we can't do as long as we allow classes. Still, the fact that basic input ranges and forward ranges inherently have different copying semantics means that they don't have quite the same problem with classes. - Jonathan M Davis
Jun 22
prev sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Thursday, 18 June 2020 at 15:55:06 UTC, jmh530 wrote:
 [snip]
Added bugzilla https://issues.dlang.org/show_bug.cgi?id=20978
Jun 25
prev sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
On Saturday, 13 June 2020 at 19:42:47 UTC, Andrei Alexandrescu 
wrote:
 I wonder how well this would work.
I've thought about this before and I think it has a lot of potential (back when we were first looking at dub, this was my preferred version scheme and everyone shot it down :( ) I believe if the new version just public imported the old version and then you add new/breaking functionality there, you could bring it all together with minimal duplication and maximum help from the compiler in making it work, including using two incompatible versions simultaneously because they'd have different mangles from different module names. I'd like to see us try to play with it.
Jun 13
prev sibling next sibling parent Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Saturday, 13 June 2020 at 03:33:14 UTC, Andrei Alexandrescu 
wrote:
 On 6/12/20 9:42 AM, 12345swordy wrote:
 On Friday, 12 June 2020 at 08:27:07 UTC, MoonlightSentinel 
 wrote:
 On Friday, 12 June 2020 at 00:24:41 UTC, 12345swordy wrote:
 Quite curious that I never see any attempt on it what so 
 ever.
I assume you are talking about DMD's test runner (https://github.com/dlang/dmd/blob/master/test/run.d). There are no concrete plan's AFAICT but I generally like the idea to replace make for the other repos as well.
I'm Sorry, I was referring to this: https://github.com/dlang/dmd/blob/master/src/build.d
That should be killed with fire. I have seldom disliked a program this much.
It's one of the best pieces of engineering in DMD :) Shall we delete dmd then? :D
Jun 13
prev sibling next sibling parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Saturday, 13 June 2020 at 03:33:14 UTC, Andrei Alexandrescu 
wrote:
 That should be killed with fire. I have seldom disliked a 
 program this much.
If anything should be killed with fire that should probably be the use of DDoc macros in dlang.org. It's a prime example of https://wiki.c2.com/?JobSecurity, but with the difference with that analogy that the only people needing that job security had already quit and now the whole thing is so messed up that no one can fix it without rewriting it from scratch.
Jun 13
next sibling parent reply David Gileadi <gileadisNOSPM gmail.com> writes:
On 6/13/20 2:05 AM, Petar Kirov [ZombineDev] wrote:
 On Saturday, 13 June 2020 at 03:33:14 UTC, Andrei Alexandrescu wrote:
 That should be killed with fire. I have seldom disliked a program this 
 much.
If anything should be killed with fire that should probably be the use of DDoc macros in dlang.org. It's a prime example of https://wiki.c2.com/?JobSecurity, but with the difference with that analogy that the only people needing that job security had already quit and now the whole thing is so messed up that no one can fix it without rewriting it from scratch.
At one time I had ambitions to search and replace the documentation to use markdown syntax instead of Ddoc macros to the extent possible. I just haven't made time for it yet, sadly. One in particular I'd like to replace is the silly $(P) macro on every single paragraph, but that would require another PR to Ddoc to do automatically create paragraphs instead of <BR>s. In my original "markdown" PR that got shot down and I didn't try to bring it back at the time.
Jun 13
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Saturday, 13 June 2020 at 15:25:07 UTC, David Gileadi wrote:
 One in particular I'd like to replace is the silly $(P) macro 
 on every single paragraph
Fun fact: this is the key issue that caused me to actually create adrdox. Lots of problems with ddoc but the horrible paragraph handling was the finishing blow.
Jun 13
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/13/20 11:33 AM, Adam D. Ruppe wrote:
 On Saturday, 13 June 2020 at 15:25:07 UTC, David Gileadi wrote:
 One in particular I'd like to replace is the silly $(P) macro on every 
 single paragraph
Fun fact: this is the key issue that caused me to actually create adrdox. Lots of problems with ddoc but the horrible paragraph handling was the finishing blow.
Yah that's awful. At a point either myself or someone else was working on automatically inserting a $(P ...) whenever a double newline was present. I think that was an easy fix.
Jun 13
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Saturday, 13 June 2020 at 18:30:31 UTC, Andrei Alexandrescu 
wrote:
 Yah that's awful. At a point either myself or someone else was 
 working on automatically inserting a $(P ...) whenever a double 
 newline was present. I think that was an easy fix.
Indeed, and ddoc actually kinda does that for .d files, it just doesn't for .dd files. Weird rules. But \n\n is not necessarily a paragraph break. Consider this: /++ First paragraph $(DIVC foo, bar ) +/ That \n\n there is immediately followed by a div tag... and should not actually have a <p> before it. It isn't completely wrong but it isn't right semantically and can lead to spacing bugs in the resulting page (it is invalid html to have a div inside a p, so it will automatically close the <p> tag before opening the <div>). So it is just a bit more complicated than it looks. My implementation to cover all the crazy edge cases I thought up is like 300 lines of code and it still isn't actually perfect (of course any time you have a 300 line buggy function there's the possibility that I just overcomplicated it and there's a much simpler, more general solution I missed).
Jun 15
prev sibling next sibling parent Dennis <dkorpel gmail.com> writes:
On Saturday, 13 June 2020 at 09:05:46 UTC, Petar Kirov 
[ZombineDev] wrote:
 If anything should be killed with fire that should probably be 
 the use of DDoc macros in dlang.org.
Out of curiosity, are there any more details on that? Like a Bugzilla issue, forum post, PR discussion etc. This complaint is new to me.
Jun 13
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/13/20 5:05 AM, Petar Kirov [ZombineDev] wrote:
 On Saturday, 13 June 2020 at 03:33:14 UTC, Andrei Alexandrescu wrote:
 That should be killed with fire. I have seldom disliked a program this 
 much.
If anything should be killed with fire that should probably be the use of DDoc macros in dlang.org. It's a prime example of https://wiki.c2.com/?JobSecurity, but with the difference with that analogy that the only people needing that job security had already quit and now the whole thing is so messed up that no one can fix it without rewriting it from scratch.
What's wrong with those? Also: don't forget that it's easy to define a battery of macros to translate to other formats. In addition to html there are translations to ebook, LaTeX, plain text, and one that I found interesting to define, which outputs the actual macros back. Or something like that. Oh, I found it: https://github.com/dlang/dlang.org/blob/master/verbatim.ddoc. Cool beans. To translate to other formats automatically or semiautomatically you could start with such a battery of macros and define them appropriately.
Jun 13
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Saturday, 13 June 2020 at 18:22:27 UTC, Andrei Alexandrescu 
wrote:
 What's wrong with those? Also: don't forget that it's easy to 
 define a battery of macros to translate to other formats.
This doesn't actually work since there's some characters with different meanings in those targets and ddoc's only answer to it is to introduce more and more macros for single characters. It is a big hassle to actually use and gives no real world benefit since using html as an intermediate format is easier anyway!
Jun 15
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/15/20 12:58 PM, Adam D. Ruppe wrote:
 On Saturday, 13 June 2020 at 18:22:27 UTC, Andrei Alexandrescu wrote:
 What's wrong with those? Also: don't forget that it's easy to define a 
 battery of macros to translate to other formats.
This doesn't actually work since there's some characters with different meanings in those targets and ddoc's only answer to it is to introduce more and more macros for single characters.
There is a way to translate individual characters before applying macros. I'm not sure of its limitations.
Jun 15
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Tuesday, 16 June 2020 at 01:15:08 UTC, Andrei Alexandrescu 
wrote:
 There is a way to translate individual characters before 
 applying macros. I'm not sure of its limitations.
That only works on code blocks and inline `code`, actually. ddoc has a misfeature of "embedded html" that specifies this bad behavior in other contexts so I wasn't able to convince the language maintainers to change it globally. https://dlang.org/spec/ddoc.html#embedded_html
Jun 15
prev sibling parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Saturday, 13 June 2020 at 03:33:14 UTC, Andrei Alexandrescu 
wrote:
 That should be killed with fire. I have seldom disliked a 
 program this much.
I think the most accurate way to classify your message is as "shitposting", which according to the (5th) definition in the Urban Dictionary [1] is: 1: The failure to make a constructive post 2: The inability to add useful information to a forum 3: Worthless overly offensive generally racists posts written in a manner which aggravates others. 4: Nrom Andrei, please use a more professional demeanor. [1]: https://www.urbandictionary.com/define.php?term=shitposting
Jun 13
next sibling parent reply Seb <seb wilzba.ch> writes:
On Saturday, 13 June 2020 at 09:24:52 UTC, Petar Kirov 
[ZombineDev] wrote:
 On Saturday, 13 June 2020 at 03:33:14 UTC, Andrei Alexandrescu 
 wrote:
 That should be killed with fire. I have seldom disliked a 
 program this much.
I think the most accurate way to classify your message is as "shitposting", which according to the (5th) definition in the Urban Dictionary [1] is: 1: The failure to make a constructive post 2: The inability to add useful information to a forum 3: Worthless overly offensive generally racists posts written in a manner which aggravates others. 4: Nrom Andrei, please use a more professional demeanor. [1]: https://www.urbandictionary.com/define.php?term=shitposting
I can't agree more. I miss the "original Andrei" :/
Jun 13
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/13/20 8:41 AM, Seb wrote:
 On Saturday, 13 June 2020 at 09:24:52 UTC, Petar Kirov [ZombineDev] wrote:
 On Saturday, 13 June 2020 at 03:33:14 UTC, Andrei Alexandrescu wrote:
 That should be killed with fire. I have seldom disliked a program 
 this much.
I think the most accurate way to classify your message is as "shitposting", which according to the (5th) definition in the Urban Dictionary [1] is: 1: The failure to make a constructive post 2: The inability to add useful information to a forum 3: Worthless overly offensive generally racists posts written in a manner which aggravates others. 4: Nrom Andrei, please use a more professional demeanor. [1]: https://www.urbandictionary.com/define.php?term=shitposting
I can't agree more. I miss the "original Andrei" :/
The original Andrei, just like today's Andrei, has an appreciation for good engineering. I didn't feel the need to add to provide detail because (a) most regulars in this forum already knew what I was going to say, and (b) nobody save for a few would share my opinion. But, I'll bite again, again to regret it. The absolute minimum - and I repeat, absolute minimum, first thing, knee-jerk thing - that one would expect from a custom build program would be that it separates the data (cmdline options, file names, environment vars, everything that's supposed to change often and easily) from the code that manipulates them and deals with things like tracking dependencies, parsing things, etc. If all is implemented in a single file I'd expect the data first, rules next, details at the end. build.d almost seems to want to make a point to be nothing like that. * Available rules (rootRules) nicely appear on line 40. * Then, we have main() dealing with a lot of stuff. (Why?) * The rules using makeRule are a confusing mix of code and data starting at line 249, after the reader has browsed through a bunch of implementation details. (An interesting side question would be what language features would make such things easier to write; e.g. a mixin string in conjunction with a DSL that implements a small subset of make may be nicer. But also becomes ironic - we started by going away from make.) * Configuration file stuff is at lines 297-315. * Hundreds of lines of rules follow. It is natural to ask oneself to what extent they improve on makefile (or other build tools') syntax. I mean is this something easy on the eyes? Did build.d attain its objective? 582 /// BuildRule to generate man pages 583 alias man = makeRule!((builder, rule) { 584 alias genMan = methodInit!(BuildRule, (genManBuilder, genManRule) => genManBuilder 585 .target(env["G"].buildPath("gen_man")) 586 .sources([ 587 dmdRepo.buildPath("docs", "gen_man.d"), 588 env["D"].buildPath("cli.d")]) 589 .command([ 590 env["HOST_DMD_RUN"], 591 "-I" ~ srcDir, 592 "-of" ~ genManRule.target] 593 ~ flags["DFLAGS"] 594 ~ genManRule.sources) 595 .msg(genManRule.command.join(" ")) 596 ); 597 598 const genManDir = env["GENERATED"].buildPath("docs", "man"); 599 alias dmdMan = methodInit!(BuildRule, (dmdManBuilder, dmdManRule) => dmdManBuilder 600 .target(genManDir.buildPath("man1", "dmd.1")) 601 .deps([genMan, directoryRule(dmdManRule.target.dirName)]) 602 .msg("(GEN_MAN) " ~ dmdManRule.target) 603 .commandFunction(() { 604 writeText(dmdManRule.target, genMan.target.execute.output); 605 }) 606 ); 607 builder 608 .name("man") 609 .description("Generate and prepare man files") 610 .deps([dmdMan].chain( 611 "man1/dumpobj.1 man1/obj2asm.1 man5/dmd.conf.5".split 612 .map!(e => methodInit!(BuildRule, (manFileBuilder, manFileRule) => manFileBuilder 613 .target(genManDir.buildPath(e)) 614 .sources([dmdRepo.buildPath("docs", "man", e)]) 615 .deps([directoryRule(manFileRule.target.dirName)]) 616 .commandFunction(() => copyAndTouch(manFileRule.sources[0], manFileRule.target)) 617 .msg("copy '%s' to '%s'".format(manFileRule.sources[0], manFileRule.target)) 618 )) 619 ).array); 620 }); * The source files are hardcoded somewhere starting at line 1160. I wish I was kidding. * A lot more support code follows, which is fine for a single-file build tool. I wish this were a two-files affair a la build.d and buildinfo.d, or if single file have an obvious separation: /**************************************************************** IMPLEMENTATION FOLLOWS ****************************************************************/
Jun 13
next sibling parent reply Dennis <dkorpel gmail.com> writes:
On Saturday, 13 June 2020 at 18:56:55 UTC, Andrei Alexandrescu 
wrote:
 I didn't feel the need to add to provide detail because (a) 
 most regulars in this forum already knew what I was going to 
 say, and (b) nobody save for a few would share my opinion.

 But, I'll bite again, again to regret it.
I, for one, thought it was an interesting little write-up. Didn't expect it, my guess was that you were against the idea of using D to build something instead of a DSL, not the implementation.
Jun 13
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/13/20 6:52 PM, Dennis wrote:
 On Saturday, 13 June 2020 at 18:56:55 UTC, Andrei Alexandrescu wrote:
 I didn't feel the need to add to provide detail because (a) most 
 regulars in this forum already knew what I was going to say, and (b) 
 nobody save for a few would share my opinion.

 But, I'll bite again, again to regret it.
I, for one, thought it was an interesting little write-up. Didn't expect it, my guess was that you were against the idea of using D to build something instead of a DSL, not the implementation.
I should add - the fact that dmd needs to be installed in order to build dmd is the proverbial insult added to the injury. Of course that ruined the carefully constructed AUTO_BOOTSTRAP option that allows building dmd on a fresh system. I should also add - unless I'm looking at the wrong version, the old posix.mak has 654 lines. build.d has 1932 lines. But build.d also supplants the Windows 32/64 makefiles (589/57 lines), so the size is not way bigger. But that begs the question - given its liabilities, by what metric is build.d an improvement?
Jun 13
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Saturday, 13 June 2020 at 23:53:25 UTC, Andrei Alexandrescu 
wrote:
 On 6/13/20 6:52 PM, Dennis wrote:
 On Saturday, 13 June 2020 at 18:56:55 UTC, Andrei Alexandrescu 
 wrote:
 I didn't feel the need to add to provide detail because (a) 
 most regulars in this forum already knew what I was going to 
 say, and (b) nobody save for a few would share my opinion.

 But, I'll bite again, again to regret it.
I, for one, thought it was an interesting little write-up. Didn't expect it, my guess was that you were against the idea of using D to build something instead of a DSL, not the implementation.
I should add - the fact that dmd needs to be installed in order to build dmd is the proverbial insult added to the injury. Of course that ruined the carefully constructed AUTO_BOOTSTRAP option that allows building dmd on a fresh system. I should also add - unless I'm looking at the wrong version, the old posix.mak has 654 lines. build.d has 1932 lines. But build.d also supplants the Windows 32/64 makefiles (589/57 lines), so the size is not way bigger. But that begs the question - given its liabilities, by what metric is build.d an improvement?
You used to have to update diffrent makefiles when adding a new file the build Have you used the windows makefiles? They didn't work without modification. I do agree that build.d takes to long to build because of all the phobos-ness in it. But if I want to debug it I can, at least. whereas with those makefiles which have to work with digital mars make. it's way harder.
Jun 13
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/13/20 8:00 PM, Stefan Koch wrote:
 On Saturday, 13 June 2020 at 23:53:25 UTC, Andrei Alexandrescu wrote:
 On 6/13/20 6:52 PM, Dennis wrote:
 On Saturday, 13 June 2020 at 18:56:55 UTC, Andrei Alexandrescu wrote:
 I didn't feel the need to add to provide detail because (a) most 
 regulars in this forum already knew what I was going to say, and (b) 
 nobody save for a few would share my opinion.

 But, I'll bite again, again to regret it.
I, for one, thought it was an interesting little write-up. Didn't expect it, my guess was that you were against the idea of using D to build something instead of a DSL, not the implementation.
I should add - the fact that dmd needs to be installed in order to build dmd is the proverbial insult added to the injury. Of course that ruined the carefully constructed AUTO_BOOTSTRAP option that allows building dmd on a fresh system. I should also add - unless I'm looking at the wrong version, the old posix.mak has 654 lines. build.d has 1932 lines. But build.d also supplants the Windows 32/64 makefiles (589/57 lines), so the size is not way bigger. But that begs the question - given its liabilities, by what metric is build.d an improvement?
You used to have to update diffrent makefiles when adding a new file the build
Awesome, now I only need to do surgery on build.d in an obscure function at line 1160 :o). I'm not saying what was there was better. I'm just asking why there's so little progress.
 Have you used the windows makefiles?
 They didn't work without modification.
 
 I do agree that build.d takes to long to build because of all the 
 phobos-ness in it.
 But if I want to debug it I can, at least.
 
 whereas with those makefiles which have to work with digital mars make.
 it's way harder.
I've often said gmake should be used for Windows to build dmd. Now we require dmd to build dmd... And this time is on us. We can't blame the arcane make syntax. We got to define it all, and defined it to be even more arcane. Where. Is. The. Progress.
Jun 13
next sibling parent reply Dennis <dkorpel gmail.com> writes:
On Sunday, 14 June 2020 at 00:22:29 UTC, Andrei Alexandrescu 
wrote:
 Now we require dmd to build dmd...
We do anyway ever since the code base got converted from C++ to D.
Jun 13
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/13/20 8:59 PM, Dennis wrote:
 On Sunday, 14 June 2020 at 00:22:29 UTC, Andrei Alexandrescu wrote:
 Now we require dmd to build dmd...
We do anyway ever since the code base got converted from C++ to D.
As I mentioned, that was done carefully and beautifully by an AUTO_BOOTSTRAP option (thanks Martin Nowak). Now build.d is the first thing that needs to run, so the entire AUTO_BOOTSTRAP thing is broken. Did I mention I appreciate good engineering?
Jun 13
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Sunday, 14 June 2020 at 01:26:02 UTC, Andrei Alexandrescu 
wrote:
 On 6/13/20 8:59 PM, Dennis wrote:
 On Sunday, 14 June 2020 at 00:22:29 UTC, Andrei Alexandrescu 
 wrote:
 Now we require dmd to build dmd...
We do anyway ever since the code base got converted from C++ to D.
As I mentioned, that was done carefully and beautifully by an AUTO_BOOTSTRAP option (thanks Martin Nowak). Now build.d is the first thing that needs to run, so the entire AUTO_BOOTSTRAP thing is broken. Did I mention I appreciate good engineering?
make -f posix.mak AUTO_BOOTSTRAP is still a thing.
Jun 13
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/13/20 9:30 PM, Stefan Koch wrote:
 On Sunday, 14 June 2020 at 01:26:02 UTC, Andrei Alexandrescu wrote:
 On 6/13/20 8:59 PM, Dennis wrote:
 On Sunday, 14 June 2020 at 00:22:29 UTC, Andrei Alexandrescu wrote:
 Now we require dmd to build dmd...
We do anyway ever since the code base got converted from C++ to D.
As I mentioned, that was done carefully and beautifully by an AUTO_BOOTSTRAP option (thanks Martin Nowak). Now build.d is the first thing that needs to run, so the entire AUTO_BOOTSTRAP thing is broken. Did I mention I appreciate good engineering?
make -f posix.mak AUTO_BOOTSTRAP is still a thing.
Not when I tried it: https://forum.dlang.org/post/r5e8uj$i1b$1 digitalmars.com
Jun 13
prev sibling parent reply Max Samukha <maxsamukha gmail.com> writes:
On Sunday, 14 June 2020 at 00:22:29 UTC, Andrei Alexandrescu 
wrote:

 Where. Is. The. Progress.
Progress would be to fix the makefiles so that they are usable out-of-the-box with the newer versions of VS. This is abysmal: VCDIR=\Program Files (x86)\Microsoft Visual Studio 10.0\VC SDKDIR=\Program Files (x86)\Microsoft SDKs\Windows\v7.0A ... CC=$(VCDIR)\bin\amd64\cl LD=$(VCDIR)\bin\amd64\link AR=$(VCDIR)\bin\amd64\lib
Jun 13
parent Seb <seb wilzba.ch> writes:
On Sunday, 14 June 2020 at 06:07:40 UTC, Max Samukha wrote:
 On Sunday, 14 June 2020 at 00:22:29 UTC, Andrei Alexandrescu 
 wrote:

 Where. Is. The. Progress.
Progress would be to fix the makefiles so that they are usable out-of-the-box with the newer versions of VS. This is abysmal: VCDIR=\Program Files (x86)\Microsoft Visual Studio 10.0\VC SDKDIR=\Program Files (x86)\Microsoft SDKs\Windows\v7.0A ... CC=$(VCDIR)\bin\amd64\cl LD=$(VCDIR)\bin\amd64\link AR=$(VCDIR)\bin\amd64\lib
For DMD, the Makefiles are only there for legacy CI reasons or AUTO_BOOTSTRAP and they might even be fully removed soonish. So at least to build DMD, everything should work out of the box on any platform when executing `./src/build.d` (or `rdmd src\build.d` on Windows) and there should be no need to fight with the hard-coded paths to Visual Studio 10. If not, please open issues on Bugzilla (and/or report here).
Jun 14
prev sibling next sibling parent Avrina <avrina12309412342 gmail.com> writes:
On Saturday, 13 June 2020 at 18:56:55 UTC, Andrei Alexandrescu 
wrote:
 * Hundreds of lines of rules follow. It is natural to ask 
 oneself to what extent they improve on makefile (or other build 
 tools') syntax. I mean is this something easy on the eyes? Did 
 build.d attain its objective?

  582  /// BuildRule to generate man pages
    583  alias man = makeRule!((builder, rule) {
    584      alias genMan = methodInit!(BuildRule, 
 (genManBuilder, genManRule) => genManBuilder
    585          .target(env["G"].buildPath("gen_man"))
    586          .sources([
    587              dmdRepo.buildPath("docs", "gen_man.d"),
    588              env["D"].buildPath("cli.d")])
    589          .command([
    590              env["HOST_DMD_RUN"],
    591              "-I" ~ srcDir,
    592              "-of" ~ genManRule.target]
    593              ~ flags["DFLAGS"]
    594              ~ genManRule.sources)
    595          .msg(genManRule.command.join(" "))
    596      );
    597
    598      const genManDir = 
 env["GENERATED"].buildPath("docs", "man");
    599      alias dmdMan = methodInit!(BuildRule, 
 (dmdManBuilder, dmdManRule) => dmdManBuilder
    600          .target(genManDir.buildPath("man1", "dmd.1"))
    601          .deps([genMan, 
 directoryRule(dmdManRule.target.dirName)])
    602          .msg("(GEN_MAN) " ~ dmdManRule.target)
    603          .commandFunction(() {
    604              writeText(dmdManRule.target, 
 genMan.target.execute.output);
    605          })
    606      );
    607      builder
    608      .name("man")
    609      .description("Generate and prepare man files")
    610      .deps([dmdMan].chain(
    611          "man1/dumpobj.1 man1/obj2asm.1 
 man5/dmd.conf.5".split
    612          .map!(e => methodInit!(BuildRule, 
 (manFileBuilder, manFileRule) => manFileBuilder
    613              .target(genManDir.buildPath(e))
    614              .sources([dmdRepo.buildPath("docs", "man", 
 e)])
    615              
 .deps([directoryRule(manFileRule.target.dirName)])
    616              .commandFunction(() => 
 copyAndTouch(manFileRule.sources[0], manFileRule.target))
    617              .msg("copy '%s' to 
 '%s'".format(manFileRule.sources[0], manFileRule.target))
    618          ))
    619      ).array);
    620  });
I agree, "chaining" is touted as one of the benefits of D, but whenever I see people using them they tend to be these types of monstrosities with many inefficiencies. The developers tend to not even know what's going on as most of the logic is hidden.
Jun 13
prev sibling parent reply Seb <seb wilzba.ch> writes:
On Saturday, 13 June 2020 at 18:56:55 UTC, Andrei Alexandrescu 
wrote:
 The original Andrei, just like today's Andrei, has an 
 appreciation for good engineering. I didn't feel the need to 
 add to provide detail because (a) most regulars in this forum 
 already knew what I was going to say, and (b) nobody save for a 
 few would share my opinion.

 But, I'll bite again, again to regret it.
I'll reply with similar regrets, because I agree that this discussion isn't helping anyone. tl;dr: I agree with you that build.d isn't the ideal engineering piece that D was supposed to enable, but it did solve a few real problems and using D without dependency was really the only option to move forward: In Detail: 1) DigitalMars Make was a massive real problem (and still is for Druntime+Phobos) 2) Duplication into many similar Makefiles with mismatching definitions (posix.mak, win32.mak, win64.mak, osmodel.mak, ...) 3) There were many things we couldn't model properly in Make (e.g. version detection shell script, documentation build, ...) 4) Especially the Windows Makefiles were missing a lot of target and options (e.g. no LTO/PGO builds, no library builds, ...) https://github.com/dlang/dmd/blob/v2.080.0/src/posix.mak https://github.com/dlang/dmd/blob/v2.080.0/src/win32.mak https://github.com/dlang/dmd/blob/v2.080.0/src/win64.mak 5) You and Walter repeatedly struck down all attempts to use any other build tool (like e.g. reggae for Phobos - https://github.com/dlang/phobos/pull/4194) 6) The Windows Makefiles looked liked this: --- $G/nteh.obj : $C\rtlsym.h $C\nteh.c $(CC) -c -o$ $(MFLAGS) $C\nteh $G/os.obj : $C\os.c $(CC) -c -o$ $(MFLAGS) $C\os $G/out.obj : $C\out.c $(CC) -c -o$ $(MFLAGS) $C\out $G/outbuf.obj : $C\outbuf.h $C\outbuf.c $(CC) -c -o$ $(MFLAGS) $C\outbuf $G/pdata.obj : $C\pdata.c $(CC) -c -o$ $(MFLAGS) $C\pdata $G/ph2.obj : $C\ph2.c $(CC) -c -o$ $(MFLAGS) $C\ph2 --- 7) Also, complexity in general increased over time as C++ header generation or CXX compatibility tests got added. There are a couple of other new features that are quite helpful for CIs (like e.g. the very detailed failure message on build failures). In build.d there definitely are a few relicts from the Makefiles times as it was a lot easier to do a step-by-step migration of the targets than migrating everything in one bulk, so I'm not denying that the build script source code could be refactored and improved, but so could be almost any code in the D ecosystem. Lastly, if you really want to know why build.d grew from 250 lines initially to its current size, the answer is rather simple. It's because of many, many shortcomings and bugs in Druntime and Phobos. As a prime example I encourage you to read through is: https://github.com/dlang/dmd/pull/10611 (GitHub hides most of the discussion by default)
 * Then, we have main() dealing with a lot of stuff. (Why?)
Not sure, what's a "a lot of stuff" for you, but that's the main method as of now: --- int main(string[] args) { try { runMain(args); return 0; } catch (BuildException e) { // Ensure paths are relative to the root directory // s.t. error messages are clickable in most IDE's writeln(e.msg.replace(buildPath("dmd", ""), buildPath("src", "dmd", ""))); return 1; } } --- To be fair, runMain does more things (gettings arguments + starting thread pool) and of course, the showHelp() could have easily been moved outside, but I really don't see how this is a critical problem for D? Especially considering how big main() is at "your" rdmd: https://github.com/dlang/tools/blob/master/rdmd.d#L62
 * The rules using makeRule are a confusing mix of code and data 
 starting at line 249, after the reader has browsed through a 
 bunch of implementation details.
1) Those are all rules. The implementation details are the rules... 2) The "reader" has a nice overview of all rules and targets at the beginning (before implementation) 3)) A typical "reader" is much more likely to execute ./build.d -h (or --help) than reading the source
 An interesting side question  would be what language features
 would make such things easier to write; e.g. a mixin string
 in conjunction with a DSL that implements a small subset of 
 make may be nicer.
Maybe a standard library with isn't riddled with hundreds of bugs and old/outdated/unusable code?
 But also becomes ironic - we started by going away from make.)
It's not ironic. WE CAN'T USE MAKE ON WINDOWS.
 * Hundreds of lines of rules follow.
Most of these rules are to support ugly things like e.g. the complicated ddoc or to workaround shortcomings in the standard library (e.g. no safe cross-platform scheduler).
 Did build.d attain its objective?
Absolutely. One big example is that people can now build DMD on Windows (this was _very_ hard/near to impossible before).
 * The source files are hardcoded somewhere starting at line 
 1160. I wish I was kidding.
build.d used globbing initially. It was removed, because Walter explicitly requested hard-coding. I can't find the PR with the request at the moment, but here's one of the PRs which removed the globbing in favor of hard-coding: https://github.com/dlang/dmd/pull/10212
 * A lot more support code follows, which is fine for a 
 single-file build tool. I wish this were a two-files affair a 
 la build.d and buildinfo.d, or if single file have an obvious 
 separation:
While we could have done a separation for rdmd which you struck done a while ago [1] which btw lead to the only two people interested in contributing to rdmd loosing their interest. Now, to answer your question fully I have to add that one of those two people (Jonathan Marler) has since then decided to create a better rdmd: rund [2]. Rund fixes one of the main shortcomings of rdmd (the need to compile a file twice with dmd and thus having an almost doubled run time) and adds a couple of other handy features like being able to specify execution flags, e.g. --- //!importPath <path> --- If rdmd would have such a feature, we could split build.d into multiple files. Unfortunately, it doesn't and thus it must remain a single file as `./build.d` needs to work out of the box. [1] https://github.com/dlang/tools/pull/344 [2] https://github.com/dragon-lang/rund
 or if single file have an obvious separation:
The separation is already there [1] , but I guess one could make it more obious. [1] https://github.com/dlang/dmd/blob/master/src/build.d#L768
 I've often said gmake should be used for Windows to build dmd. 
 Now we require dmd to build dmd...
Yes, how else is one going to build D code since 2.067?
 Of course that ruined the carefully constructed AUTO_BOOTSTRAP 
 option that allows building dmd on a fresh system.
Nope. In fact, AUTO_BOOTSTRAP is still being used by some of our own CIs, e.g. Buildkite still does ``` make -f posix.mak AUTO_BOOTSTRAP=1 ``` Some other do this to get an active D compiler in your environment: ``` source $(curl https://dlang.org/install.sh | bash -s install -a) ```
 Not when I tried it: 
 https://forum.dlang.org/post/r5e8uj$i1b$1 digitalmars.com
To be fair that's not a fault of build.d. You aren't able to compile anything with an outdated compiler on OSX. For details see some of the responses to your thread: https://forum.dlang.org/post/rimiuawxnzobcgfuwxez forum.dlang.org https://forum.dlang.org/post/zvsjvhvgprhgacybvctb forum.dlang.org
Jun 14
next sibling parent reply Bruce Carneal <bcarneal gmail.com> writes:
On Sunday, 14 June 2020 at 16:07:16 UTC, Seb wrote:
 On Saturday, 13 June 2020 at 18:56:55 UTC, Andrei Alexandrescu 
 wrote:
 The original Andrei, just like today's Andrei, has an 
 appreciation for good engineering. I didn't feel the need to 
 add to provide detail because (a) most regulars in this forum 
 already knew what I was going to say, and (b) nobody save for 
 a few would share my opinion.

 But, I'll bite again, again to regret it.
I'll reply with similar regrets, because I agree that this discussion isn't helping anyone.
It's helping me. [big snip of clarifying context and concise refutations of many earlier assertions] Thanks Seb, for the context, the refutations, and for the references to alternative tools (rund, reggae).
Jun 14
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/14/20 1:05 PM, Bruce Carneal wrote:
 On Sunday, 14 June 2020 at 16:07:16 UTC, Seb wrote:
 On Saturday, 13 June 2020 at 18:56:55 UTC, Andrei Alexandrescu wrote:
 The original Andrei, just like today's Andrei, has an appreciation 
 for good engineering. I didn't feel the need to add to provide detail 
 because (a) most regulars in this forum already knew what I was going 
 to say, and (b) nobody save for a few would share my opinion.

 But, I'll bite again, again to regret it.
I'll reply with similar regrets, because I agree that this discussion isn't helping anyone.
It's helping me. [big snip of clarifying context and concise refutations of many earlier assertions] Thanks Seb, for the context, the refutations, and for the references to alternative tools (rund, reggae).
Most of said refutations are reducible to simple misunderstandings. A few are due to my oversights. A few are Seb's. My core point stays: build.d is not Good Work(tm), and can't be talked into it. It must be worked into it. I don't want the makefiles back. I want Good Work, which build.d has a good way to go toward. A good part of my critique can be addressed by refactoring build.d in the ways I suggested. Another part of the critique would be achieved by a pass through its design, with an eye for making dependency syntax tolerable.
Jun 14
parent Bruce Carneal <bcarneal gmail.com> writes:
On Sunday, 14 June 2020 at 17:40:04 UTC, Andrei Alexandrescu 
wrote:
 On 6/14/20 1:05 PM, Bruce Carneal wrote:
 On Sunday, 14 June 2020 at 16:07:16 UTC, Seb wrote:
 On Saturday, 13 June 2020 at 18:56:55 UTC, Andrei
[big snip of clarifying context and concise refutations of many earlier assertions] Thanks Seb, for the context, the refutations, and for the references to alternative tools (rund, reggae).
Most of said refutations are reducible to simple misunderstandings. A few are due to my oversights. A few are Seb's. My core point stays: build.d is not Good Work(tm), and can't be talked into it. It must be worked into it. I don't want the makefiles back. I want Good Work, which build.d has a good way to go toward. A good part of my critique can be addressed by refactoring build.d in the ways I suggested. Another part of the critique would be achieved by a pass through its design, with an eye for making dependency syntax tolerable.
I'm not a domain expert on this but from my earlier sight-reading of the code it sure looks like a rework/refactor/rewrite candidate to me as well. More importantly Seb, a domain expert, appears to agree with you on this. Such an agreement would make this a part of the well recognized bigger challenge: engaging qualified programmers from a mostly-volunteer pool to address known problems.
Jun 14
prev sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Sunday, 14 June 2020 at 16:07:16 UTC, Seb wrote:
 While we could have done a separation for rdmd which you struck 
 done a while ago [1] which btw lead to the only two people 
 interested in contributing to rdmd loosing their interest.

 ...

 [1] https://github.com/dlang/tools/pull/344
It's a bit tangential to the main thrust of discussion here, but I'd just like it on the record that the decision on that PR had nothing to do with why I didn't continue to contribute to rdmd. I just had a lot of other things keeping me busy (and still do). Andrei and I had a very constructive conversation at DConf later that year where he suggested, and I agreed, that the way forward with rdmd's issues was not to refactor rdmd but to bring more of its functionality inside the D frontend (of which the main things would presumably be the search for the files to compile, and the running of the compiled executable). I don't know what if any progress has been made in that respect but it still sounds the most reasonable way to clean things up and avoid duplication of effort between the codebases. BTW, I think it is important to listen carefully to Andrei's broader feedback here. A lot of the reason why I got hands on with rdmd was because I was seeing a lot of what I felt were problematic development decisions, and I wanted to clean up the codebase and put in place much more rigorous tests to help avoid those problems in future. I know it's not easy to hear feedback like this from someone who isn't hands-on at the coalface alongside you, but it really feels like far too often work gets accepted because "something must be done!" and in order to encourage the person who is stepping up to do the work, rather than asking the harder question of, is it being done the right way, and is that work really producing the right long-term outcome.
Jun 23
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/13/20 5:24 AM, Petar Kirov [ZombineDev] wrote:
 On Saturday, 13 June 2020 at 03:33:14 UTC, Andrei Alexandrescu wrote:
 That should be killed with fire. I have seldom disliked a program this 
 much.
I think the most accurate way to classify your message is as "shitposting", which according to the (5th) definition in the Urban Dictionary [1] is: 1: The failure to make a constructive post 2: The inability to add useful information to a forum 3: Worthless overly offensive generally racists posts written in a manner which aggravates others. 4: Nrom Andrei, please use a more professional demeanor. [1]: https://www.urbandictionary.com/define.php?term=shitposting
Nice going. Particularly lovely is the weaseling of loaded words that make it all look more righteous.
Jun 13