www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Phobos' slow descent into madness

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
So I was looking at this overly beefy compilation with an eye on what 
can cause it take so much memory and time.

Was using -vtemplates and was surprised to see tens of thousands (I mean 
literally: over 45,000 of them) of distinct instantiations of the 
obscure template AliasThisTypeOf.

What the heck is that? Turns out it's a private symbol (that for some 
reason turns out to be duplicated in druntime and phobos). That private 
symbol, in turn, is used in a suite of desperate templates of which 
drabness is only overcome by their mind-numbing repetitiveness:

BooleanTypeOf
IntegralTypeOf
FloatingPointTypeOf
CharTypeOf
StaticArrayTypeOf
DynamicArrayTypeOf
AssocArrayTypeOf

What are these used for? Glad you asked! All these support "alias this" 
for the std.format.format function.

In turn, format turns out to be used, often gratuitously, in very, very 
many places during compilation, particularly in std.traits itself (as an 
aside: format("%s[]")? Really? What happened with good old string 
concatenation?). And that creates an ungodly amount of template 
instantiations. See for yourselves: https://run.dlang.io/is/6j9WI2

But the main problem is the Saturnism that Phobos seems to be suffering 
from. A slow descent into madness.

I do understand the boiling frog slipping on the slippery slope toward 
the road to hell paved with whatever. I do. Heck, I even added a slab 
here and there. But if we look today at the code at 
https://github.com/dlang/phobos/blob/master/std/traits.d#L5819 with the 
sobriety and clearheadedness of a priest on a Sunday morning, the only 
reaction we should have is: "This madness has gotten too far and we must 
stop and turn around RIGHT NOW."

The code in the standard library represents the language, and how we'd 
like people to do things in the language. Sure, we may err on the side 
of optimizing things and throwing in some low-level code for efficiency 
purposes. But this code is not optimization; not algorithms; nothing of 
interest, really. It's mad scaffolding around a temple that should not 
have been built in the first place.

The madness has to stop.
May 07
next sibling parent reply MoonlightSentinel <moonlightsentinel disroot.org> writes:
On Friday, 7 May 2021 at 20:47:57 UTC, Andrei Alexandrescu wrote:
 In turn, format turns out to be used, often gratuitously, in 
 very, very many places during compilation, particularly in 
 std.traits itself (as an aside: format("%s[]")? Really? What 
 happened with good old string concatenation?).
Except that string concatenation doesn't work with callback / output range based `toString` implementations.
May 07
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On Friday, 7 May 2021 at 22:36:09 UTC, MoonlightSentinel wrote:
 On Friday, 7 May 2021 at 20:47:57 UTC, Andrei Alexandrescu 
 wrote:
 In turn, format turns out to be used, often gratuitously, in 
 very, very many places during compilation, particularly in 
 std.traits itself (as an aside: format("%s[]")? Really? What 
 happened with good old string concatenation?).
Except that string concatenation doesn't work with callback / output range based `toString` implementations.
Point taken, but I was referring to compile-time uses where that is practically never. There are zero instances of that case here: https://github.com/dlang/phobos/pull/8052. Almost in all cases the code that replaces calls to format() is shorter and simpler.
May 07
prev sibling next sibling parent reply SealabJaster <sealabjaster gmail.com> writes:
On Friday, 7 May 2021 at 20:47:57 UTC, Andrei Alexandrescu wrote:
 ...
With all the nit-picking you've (thankfully) been doing recently, would it be worth writing up "Andrei's Ambitious Amendments to Phobos", a.k.a some kind of action plan you'd personally like to see going forward? I ask this as you have a very keen eye for things like this, and considering your position people are likely to listen and discuss about what you'd like to see changed. Having it all separated over multiple forum posts may be ok for discussions, but to see the overall view and direction, perhaps it is best to consolidate it all together somehow? It'd certainly be quite welcome to see a push into a specific direction for either/both the language and Phobos, especially since Phobos feels like it's in some weird sort of limbo to me. Or just ignore me because I barely know what I'm talking about >xP
May 07
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On Friday, 7 May 2021 at 23:19:30 UTC, SealabJaster wrote:
 On Friday, 7 May 2021 at 20:47:57 UTC, Andrei Alexandrescu 
 wrote:
 ...
With all the nit-picking you've (thankfully) been doing recently, would it be worth writing up "Andrei's Ambitious Amendments to Phobos", a.k.a some kind of action plan you'd personally like to see going forward? I ask this as you have a very keen eye for things like this, and considering your position people are likely to listen and discuss about what you'd like to see changed. Having it all separated over multiple forum posts may be ok for discussions, but to see the overall view and direction, perhaps it is best to consolidate it all together somehow? It'd certainly be quite welcome to see a push into a specific direction for either/both the language and Phobos, especially since Phobos feels like it's in some weird sort of limbo to me. Or just ignore me because I barely know what I'm talking about
xP
This is a nice idea, thanks. I think the gating factor here is Phobos versioning, which requires language additions. Then it would be interesting to put together a white paper.
May 07
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/7/2021 4:58 PM, Andrei Alexandrescu wrote:

 This is a nice idea, thanks. I think the gating factor here is Phobos 
 versioning, which requires language additions. Then it would be interesting to 
 put together a white paper.
 
That's a great idea. Doing a white paper, though, is not gated on language improvements. We know what those improvements need to accomplish, and shouldn't affect the redesign.
May 07
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, May 07, 2021 at 11:58:06PM +0000, Andrei Alexandrescu via Digitalmars-d
wrote:
 On Friday, 7 May 2021 at 23:19:30 UTC, SealabJaster wrote:
[...]
 Having it all separated over multiple forum posts may be ok for
 discussions, but to see the overall view and direction, perhaps it
 is best to consolidate it all together somehow?
[...]
 This is a nice idea, thanks. I think the gating factor here is Phobos
 versioning, which requires language additions. Then it would be
 interesting to put together a white paper.
I think his point is to collect the current criticisms of Phobos into a single point of reference, perhaps on the wiki or something, that could serve as a list of improvements to be made. Stuff posted in the forum will inevitably get forgotten in the dusts of time. T -- Beware of bugs in the above code; I have only proved it correct, not tried it. -- Donald Knuth
May 07
next sibling parent Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Saturday, 8 May 2021 at 01:20:34 UTC, H. S. Teoh wrote:
 On Fri, May 07, 2021 at 11:58:06PM +0000, Andrei Alexandrescu 
 via Digitalmars-d wrote:
 On Friday, 7 May 2021 at 23:19:30 UTC, SealabJaster wrote:
[...]
 Having it all separated over multiple forum posts may be ok 
 for discussions, but to see the overall view and direction, 
 perhaps it is best to consolidate it all together somehow?
[...]
 This is a nice idea, thanks. I think the gating factor here is 
 Phobos versioning, which requires language additions. Then it 
 would be interesting to put together a white paper.
I think his point is to collect the current criticisms of Phobos into a single point of reference, perhaps on the wiki or something, that could serve as a list of improvements to be made. Stuff posted in the forum will inevitably get forgotten in the dusts of time. T
+1 It's the same thing at home, you need to pause and take a look. Then you see "omg I need to clean", and when you're done you never regret it. But, you gotta be careful... https://ibb.co/sH4VZcc
May 08
prev sibling parent reply Berni44 <someone somemail.com> writes:
On Saturday, 8 May 2021 at 01:20:34 UTC, H. S. Teoh wrote:
 I think his point is to collect the current criticisms of 
 Phobos into a single point of reference, perhaps on the wiki or 
 something, that could serve as a list of improvements to be 
 made.
+1 (would be nice to have forum feature to do that without adding a new post) Recently there were rumors of Phobos 2.0. I think, that would be a chance to get us out of the muds. But for that we need such a signpost. It would help us to make us run all in more or less the same direction. To a certain extend it will be possible to improve already on Phobos 1.0. But we quite often reach points, where we need deprecation cycles (or even preview-switch-and-later-deprecation-cycles) which takes quite a long time and is annoying for users too. n8sh [recently](https://github.com/dlang/phobos/pull/7775#issuecomment-831996462) pointed me to [Dear Google Cloud: Your Deprecation Policy is Killing You](https://steve-yegge.medium.com/dear-google-cloud-your-deprecation-policy-is-killin -you-ee7525dc05dc). This made me thoughtful. I would probably like to have a policy like Java has (see the article above for more details). But before starting such a thing, I'd prefer to reach something really stable (compared to stable with the meaning of changing things all two months, which for me is somewhat a self-contradiction). So what I would like to do, would be: - Let Phobos 1.0 be as is (maybe even remove the deprecation cycles). - Move over to Phobos 2.0 (e. g.) until summer 2023 (with a list of goals at hand, e.g. no auto-*coughing*, a better range/string concept, etc.). This will include several breaking changes. - End of 2022 Phobos 2.0 will be frozen: Nothing new is accepted, only bugfixes. - From summer 2023 on, Phobos 2.0 is stable. We guarantee backward compatibility forever, but things may be deprecated. - We may then start working on Phobos 2.1, which will be released in summer 2025 and frozen at end of 2024. And so on.
May 08
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/8/21 6:01 AM, Berni44 wrote:
 
 - Let Phobos 1.0 be as is (maybe even remove the deprecation cycles).
 - Move over to Phobos 2.0 (e. g.) until summer 2023 (with a list of 
 goals at hand, e.g. no auto-*coughing*, a better range/string concept, 
 etc.). This will include several breaking changes.
 - End of 2022 Phobos 2.0 will be frozen: Nothing new is accepted, only 
 bugfixes.
 - From summer 2023 on, Phobos 2.0 is stable. We guarantee backward 
 compatibility forever, but things may be deprecated.
 - We may then start working on Phobos 2.1, which will be released in 
 summer 2025 and frozen at end of 2024. And so on.
Couple of things missing here: (a) the rule of incrementalism and (b) the rule of nonduplication. The rule of incrementalism is that code can use arbitrary parts of Phobos 2.0 in conjunction with arbitrary parts of Phobos 1.0. The migration path would be as simple as progressively replacing std.somemodule with std.v2.somemodule and get code to compile. The rule of nonduplication is there should be no code in Phobos 2.0 that is identical to code in Phobos 1.0. Whenever Phobos 2.0 needs functionality of Phobos 1.0, it must be able to do so with ease. These are much more difficult to achieve than it seems. However, they are existential - we must satisfy them if we want to survive.
May 08
parent russhy <russhy gmail.com> writes:
I'm glad i'm sticking to ``core.stdc``
May 08
prev sibling parent russhy <russhy gmail.com> writes:
On Saturday, 8 May 2021 at 10:01:27 UTC, Berni44 wrote:
 On Saturday, 8 May 2021 at 01:20:34 UTC, H. S. Teoh wrote:
 I think his point is to collect the current criticisms of 
 Phobos into a single point of reference, perhaps on the wiki 
 or something, that could serve as a list of improvements to be 
 made.
+1 (would be nice to have forum feature to do that without adding a new post) Recently there were rumors of Phobos 2.0. I think, that would be a chance to get us out of the muds. But for that we need such a signpost. It would help us to make us run all in more or less the same direction. To a certain extend it will be possible to improve already on Phobos 1.0. But we quite often reach points, where we need deprecation cycles (or even preview-switch-and-later-deprecation-cycles) which takes quite a long time and is annoying for users too. n8sh [recently](https://github.com/dlang/phobos/pull/7775#issuecomment-831996462) pointed me to [Dear Google Cloud: Your Deprecation Policy is Killing You](https://steve-yegge.medium.com/dear-google-cloud-your-deprecation-policy-is-killin -you-ee7525dc05dc). This made me thoughtful. I would probably like to have a policy like Java has (see the article above for more details). But before starting such a thing, I'd prefer to reach something really stable (compared to stable with the meaning of changing things all two months, which for me is somewhat a self-contradiction). So what I would like to do, would be: - Let Phobos 1.0 be as is (maybe even remove the deprecation cycles). - Move over to Phobos 2.0 (e. g.) until summer 2023 (with a list of goals at hand, e.g. no auto-*coughing*, a better range/string concept, etc.). This will include several breaking changes. - End of 2022 Phobos 2.0 will be frozen: Nothing new is accepted, only bugfixes. - From summer 2023 on, Phobos 2.0 is stable. We guarantee backward compatibility forever, but things may be deprecated. - We may then start working on Phobos 2.1, which will be released in summer 2025 and frozen at end of 2024. And so on.
I agree with you 100%, it needs to happen, Python did that with Python 3 and got a new and prosperous life as a result Same with .NET with dotnet core Let's hope for APIs built with Allocators in mind Let's hope people focus on efficiency and simplicity I would love to contribute to such project, i already have a lot of code built with this in mind I simply wish there was some sort of signature support for structs, that would make designing APIs much easier/nicer https://gist.github.com/rikkimax/826e1c4deb531e8dd993815bf914acea
May 08
prev sibling parent reply Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Friday, 7 May 2021 at 20:47:57 UTC, Andrei Alexandrescu wrote:
 What the heck is that? Turns out it's a private symbol (that 
 for some reason turns out to be duplicated in druntime and 
 phobos). That private symbol, in turn, is used in a suite of 
 desperate templates of which drabness is only overcome by their 
 mind-numbing repetitiveness:

 BooleanTypeOf
 IntegralTypeOf
 FloatingPointTypeOf
 CharTypeOf
 StaticArrayTypeOf
 DynamicArrayTypeOf
 AssocArrayTypeOf

 What are these used for? Glad you asked! All these support 
 "alias this" for the std.format.format function.
It's worth mentioning that, according to `git blame`, these symbols were all added by k-hara in 2012 and haven't changed in any fundamental way since. So I don't think the title of this thread quite represents the recent slow decline of things in phobos. Rather the decline happended abruptly 10 years ago. A more relevant question to ask ourselves is who approved these changes in 2012.
Sep 15
parent reply Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Wednesday, 15 September 2021 at 11:56:04 UTC, Per Nordlöw 
wrote:

 A more relevant question to ask ourselves is who approved these 
 changes in 2012.
Btw, the PR [1] introducing these traits was merged by a user that is now deleted. [1] https://github.com/dlang/phobos/pull/1759
Sep 15
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/15/21 8:04 AM, Per Nordlöw wrote:
 On Wednesday, 15 September 2021 at 11:56:04 UTC, Per Nordlöw wrote:
 
 A more relevant question to ask ourselves is who approved these 
 changes in 2012.
Btw, the PR [1] introducing these traits was merged by a user that is now deleted. [1] https://github.com/dlang/phobos/pull/1759
Most likely Andrej Mitrovich, as his email is associated with the merge commit (this is back when the bot did not actually do merges). https://github.com/dlang/phobos/commit/be647d569b54eec3d283ee3e92e8d1ab2050c31f -Steve
Sep 15