www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Back in D after some absence, some perspective

reply deadalnix <deadalnix gmail.com> writes:
Hi all,

As you might know, I was very active a couple of years ago and 
then got driven away, mostly by work, family and so on. But I've 
been back for a couple of month now and I thought I'd write a 
little about what changed on larger time scale. It is not always 
easy to see this when being in D on a day to day basis, so I 
thought this would be valuable feedback for everybody.

The bad:
I ran into many bugs still, mostly in phobos. This might indicate 
that code was merged in before it was really ready and that we 
are paying back that debt, or that the language has accumulated 
too much complexity and that it is hard target these things right.

In any case, a common pattern across several of these bugs was a 
poor handling of the "empty" case. For instance 
https://github.com/dlang/phobos/pull/7788 was due to a problem 
where std.csv did not handle empty csv files and would report at 
least one row.

That should probably be added to the checklist of test case to 
have for something to be considered mergeable. Not sure it is 
automatable, but human can use their judgement to get it sorted 
out.

Another major pain point was dub. It is wonderful to get 
something up and running quickly, but then turn into a nightmare 
when it come to packaging and distribution. The root cause seems 
to be that dub tries to be too many things at once, such as a 
dependency manager and a build system.

The good:
No more ICE! At least I haven't run into any so far.

The awesome:
The support I got through all of this was first class. I don't 
know if this is because it's me and people know me or if this is 
general, but it is way better than a couple of years ago and bugs 
gets fixed way faster.

If that pace can be sustained without introducing too many 
regressions, then I doubt I'll be running into bugs on a regular 
basis.

One word of advice:
I see that new features are being added still, such as the recent 
addition of noreturn. While this is a good addition, such a case 
can be made for a bazillion other features too. I think it is 
hard too argue that D doesn't have enough feature at this time. 
In my experience, running into bugs and difficulty to integrate 
into existing systems, workflows, dependency management, and so 
on is whee it's at.

D has reached a point where it needs to heavily focus on its 
toolchain, as well as ensuring the soundness of what's there, 
nailing small inconsistencies, etc...
Mar 02
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Mar 02, 2021 at 09:05:14PM +0000, deadalnix via Digitalmars-d wrote:
[...]
 As you might know, I was very active a couple of years ago and then
 got driven away, mostly by work, family and so on. But I've been back
 for a couple of month now and I thought I'd write a little about what
 changed on larger time scale. It is not always easy to see this when
 being in D on a day to day basis, so I thought this would be valuable
 feedback for everybody.
Very valuable indeed!
 The bad:
 I ran into many bugs still, mostly in phobos.
[...]
 In any case, a common pattern across several of these bugs was a poor
 handling of the "empty" case.
[...]
 That should probably be added to the checklist of test case to have
 for something to be considered mergeable. Not sure it is automatable,
 but human can use their judgement to get it sorted out.
I think there should be an official checklist of test cases that need to be covered before a PR is merged. Of course it can't be a precise checklist because every PR is different, but without an official, written guideline, it's too easy to just forget about this or that corner case and unknowingly introduce bugs.
 Another major pain point was dub. It is wonderful to get something up
 and running quickly, but then turn into a nightmare when it come to
 packaging and distribution. The root cause seems to be that dub tries
 to be too many things at once, such as a dependency manager and a
 build system.
I've been staying away from dub precisely because of the above architectural issues, but telling people to stay away is not a viable solution. I don't know how to solve this problem without completely redesigning dub, though. It's too entrenched to change now, and we don't want to lose the ecosystem that has been painstakingly built up around it.
 The good:
 No more ICE! At least I haven't run into any so far.
Yay!
 The awesome:
 The support I got through all of this was first class. I don't know if
 this is because it's me and people know me or if this is general, but
 it is way better than a couple of years ago and bugs gets fixed way
 faster.
I think now that the foundation has hired a couple of people to handle the mundane, day-to-day chores, there's been more grease in the gears and things are running a little more smoothly now. Hopefully this will continue into the future. [...]
 D has reached a point where it needs to heavily focus on its
 toolchain, as well as ensuring the soundness of what's there, nailing
 small inconsistencies, etc...
Yeah, the small inconsistencies really need to be fixed. They may be small, minor details from a language perspective, but these minor details cause a domino effect of weird corner cases that cause pain down the road, esp. to newcomers unfamiliar with the gotchas involved. These corner cases really need to be smoothed out. Also, to mention another (relatively) recent awesome development: LDC has been supporting almost out-of-the-box cross-compilation from Linux to Windows, and I've been really enjoying that. All it takes now is just to download and unpack LDC for Linux and LDC for Windows, then add a block to the config file, and it's all set for heavy-duty cross-compilation. Also, if you ever need to interface with Java, check out Adam's jni.d. Metaprogramming awesomeness that makes it so easy to interface D with Java, I'm actually starting to lose some of my dislike for Java. :-D T -- Why waste time learning, when ignorance is instantaneous? -- Hobbes, from Calvin & Hobbes
Mar 02
prev sibling next sibling parent Dominikus Dittes Scherkl <dominikus scherkl.de> writes:
On Tuesday, 2 March 2021 at 21:05:14 UTC, deadalnix wrote:

 One word of advice:
 I see that new features are being added still, such as the 
 recent addition of noreturn. While this is a good addition, 
 such a case can be made for a bazillion other features too.
I don't see this as a new feature - it's more like a fix for a glaring hole in the type system, that now removes a few corner cases that prevented generic coding in some places. But it was of course a big enough change to require a DIP. I think this is the kind of "features" that really should go into the language: things that _remove_ corner cases. The same is true for the safe/ system variables and for type functions. In contrast to new "syntactic sugar" like named parameters or interpolated strings, which I also like but which are more debatable.
Mar 02
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
It's great to have you back!
Mar 03
prev sibling next sibling parent Dukc <ajieskola gmail.com> writes:
On Tuesday, 2 March 2021 at 21:05:14 UTC, deadalnix wrote:
 In any case, a common pattern across several of these bugs was 
 a poor handling of the "empty" case. For instance 
 https://github.com/dlang/phobos/pull/7788 was due to a problem 
 where std.csv did not handle empty csv files and would report 
 at least one row.
Easy to see. Just yesterday a range I had just done (for my own project) malfunctioned exactly because it's so easy to screw the empty range thing up.
 Another major pain point was dub. It is wonderful to get 
 something up and running quickly, but then turn into a 
 nightmare when it come to packaging and distribution. The root 
 cause seems to be that dub tries to be too many things at once, 
 such as a dependency manager and a build system.
How so? When making a distribution package, just bundle the DUB packages, or distribute `dub.selections.json` with the source code and have your package to checksum the final binary.
 The awesome:
 The support I got through all of this was first class. I don't 
 know if this is because it's me and people know me or if this 
 is general, but it is way better than a couple of years ago and 
 bugs gets fixed way faster.
Probably in no small part due to D language foundation and the people they have hired to make sure contributions keep moving. Take DIPs for instance, a few years ago the process worked only randomly, now we have DIP after DIP, with like three or so accepted per year. The process works slowly and the quality of the decisions is sometimes debatable, but no DIP will stall forever anymore because of the DIP management/maintainers.
 If that pace can be sustained without introducing too many 
 regressions, then I doubt I'll be running into bugs on a 
 regular basis.
Both the compiler and the library have even heavier test suite than a few years ago, it easily catches the vast majority of regressions we'd have otherwise. Unfortunately it is so heavy it also slows down development considerably. And there are many cases where it's difficult to be sure what's the correct behaviour. Many instabilities result just because the user code accidently relies on such loosely defined behaviour.
 One word of advice:
 I see that new features are being added still, such as the 
 recent addition of noreturn. While this is a good addition, 
 such a case can be made for a bazillion other features too. I 
 think it is hard too argue that D doesn't have enough feature 
 at this time. In my experience, running into bugs and 
 difficulty to integrate into existing systems, workflows, 
 dependency management, and so on is whee it's at.

 D has reached a point where it needs to heavily focus on its 
 toolchain, as well as ensuring the soundness of what's there, 
 nailing small inconsistencies, etc...
Those are real issues, but I don't think stagnating the development just to let the language to stabilize is any better overall. Instead we need ways for the new code to work with old code. Perhaps something like declaring the language version and compiler preview switches in module headers.
Mar 03
prev sibling next sibling parent Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Tuesday, 2 March 2021 at 21:05:14 UTC, deadalnix wrote:

. In my experience, running into bugs and
 difficulty to integrate into existing systems, workflows, 
 dependency management, and so on is whee it's at.

 D has reached a point where it needs to heavily focus on its 
 toolchain, as well as ensuring the soundness of what's there, 
 nailing small inconsistencies, etc...
9000 upvotes on this
Mar 08
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 02.03.21 22:05, deadalnix wrote:
 I think it is hard too argue that D doesn't have enough feature at this 
 time.
OTOH, there is no pattern matching, in particular you can't pattern match tuples. Therefore, it is also very hard to argue that D is feature-complete. (Has it skipped leg day?)
Mar 08
parent reply deadalnix <deadalnix gmail.com> writes:
On Monday, 8 March 2021 at 11:01:17 UTC, Timon Gehr wrote:
 On 02.03.21 22:05, deadalnix wrote:
 I think it is hard too argue that D doesn't have enough 
 feature at this time.
OTOH, there is no pattern matching, in particular you can't pattern match tuples. Therefore, it is also very hard to argue that D is feature-complete. (Has it skipped leg day?)
While pattern matching would be nice, I place soundness above it. Plus, if you look at the SDC codebase for instance, you'll see that you can go pretty far as a library.
Mar 08
next sibling parent Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Monday, 8 March 2021 at 15:39:52 UTC, deadalnix wrote:
 On Monday, 8 March 2021 at 11:01:17 UTC, Timon Gehr wrote:
 On 02.03.21 22:05, deadalnix wrote:
 I think it is hard too argue that D doesn't have enough 
 feature at this time.
OTOH, there is no pattern matching, in particular you can't pattern match tuples. Therefore, it is also very hard to argue that D is feature-complete. (Has it skipped leg day?)
While pattern matching would be nice, I place soundness above it. Plus, if you look at the SDC codebase for instance, you'll see that you can go pretty far as a library.
Oh, have you begun working on SDC again? ☀️
Mar 08
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 08.03.21 16:39, deadalnix wrote:
 On Monday, 8 March 2021 at 11:01:17 UTC, Timon Gehr wrote:
 On 02.03.21 22:05, deadalnix wrote:
 I think it is hard too argue that D doesn't have enough feature at 
 this time.
OTOH, there is no pattern matching, in particular you can't pattern match tuples. Therefore, it is also very hard to argue that D is feature-complete. (Has it skipped leg day?)
While pattern matching would be nice, I place soundness above it.
For me it's not even a close contender. There's no soundness now and soundness is much harder to achieve than (or even without) obvious fundamental features. In comparison, soundness almost seems like a lost battle at this point.
 Plus, if you look at the SDC codebase for instance, you'll see that you can go 
 pretty far as a library.
auto firstThingSecondThing=query!((ref foo)=>tuple(foo.a, foo.b))(data); auto firstThing=firstThingSecondThing[0], secondThing=firstThingSecondThing[1]; Huge pain. auto (firstThing,secondThing)=query!((ref foo)=>(foo.a, foo.b))(data); Much better. Whatever string mixin hack you can come up with is not good enough.
Mar 08
parent reply Max Haughton <maxhaton gmail.com> writes:
On Monday, 8 March 2021 at 17:10:38 UTC, Timon Gehr wrote:
 On 08.03.21 16:39, deadalnix wrote:
 On Monday, 8 March 2021 at 11:01:17 UTC, Timon Gehr wrote:
 On 02.03.21 22:05, deadalnix wrote:
 I think it is hard too argue that D doesn't have enough 
 feature at this time.
OTOH, there is no pattern matching, in particular you can't pattern match tuples. Therefore, it is also very hard to argue that D is feature-complete. (Has it skipped leg day?)
While pattern matching would be nice, I place soundness above it.
For me it's not even a close contender. There's no soundness now and soundness is much harder to achieve than (or even without) obvious fundamental features. In comparison, soundness almost seems like a lost battle at this point.
 Plus, if you look at the SDC codebase for instance, you'll see 
 that you can go pretty far as a library.
auto firstThingSecondThing=query!((ref foo)=>tuple(foo.a, foo.b))(data); auto firstThing=firstThingSecondThing[0], secondThing=firstThingSecondThing[1]; Huge pain. auto (firstThing,secondThing)=query!((ref foo)=>(foo.a, foo.b))(data); Much better. Whatever string mixin hack you can come up with is not good enough.
I agree. I also think the weird tuple situation we have now has to look bad from afar, e.g. we have Phobos tuples which aren't great, dmd tuples which are weird, AliasSeq, T... etc. Furthermore, I read your proto-DIP about Tuples, and I think it's a good step forward, but it strikes me that maybe we should just go the whole way and have first-class tuples (Even just as a direct rewrite to a struct would be a step forward in terms of allowing the kind of unpacking syntax that I would hope almost everyone wants).
Mar 08
parent reply Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Monday, 8 March 2021 at 18:02:35 UTC, Max Haughton wrote:
 On Monday, 8 March 2021 at 17:10:38 UTC, Timon Gehr wrote:
 [...]
I agree. I also think the weird tuple situation we have now has to look bad from afar, e.g. we have Phobos tuples which aren't great, dmd tuples which are weird, AliasSeq, T... etc. Furthermore, I read your proto-DIP about Tuples, and I think it's a good step forward, but it strikes me that maybe we should just go the whole way and have first-class tuples (Even just as a direct rewrite to a struct would be a step forward in terms of allowing the kind of unpacking syntax that I would hope almost everyone wants).
Yeah, soundness is more important than features imo
Mar 09
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 09.03.21 09:32, Imperatorn wrote:
 On Monday, 8 March 2021 at 18:02:35 UTC, Max Haughton wrote:
 On Monday, 8 March 2021 at 17:10:38 UTC, Timon Gehr wrote:
 [...]
I agree. I also think the weird tuple situation we have now has to look bad from afar, e.g. we have Phobos tuples which aren't great, dmd tuples which are weird, AliasSeq, T... etc. Furthermore, I read your proto-DIP about Tuples, and I think it's a good step forward, but it strikes me that maybe we should just go the whole way and have first-class tuples (Even just as a direct rewrite to a struct would be a step forward in terms of allowing the kind of unpacking syntax that I would hope almost everyone wants).
Yeah, soundness is more important than features imo
It sounds nice, but I think it is just not true, and it has never been the philosophy driving D's design. Right now, D is unsound in various ways, but we like to use it anyway, because it has features. Soundness is important, but I suffer more from lack of features than lack of full soundness. E.g., philosophically, assertion failures causing UB in safe code is a terrible, unsound design, but in practice I can avoid it and I have given up on trying to make the point to Walter. On the topic of assertion errors: `in` contract inheritance catches AssertError. Is this an issue that's more important than built-in tuples just because it is a case of blatant unsoundness? Not even close. I also think it is quite safe to assume that DMD will never implement any consistent semantics for the interaction of forward references and introspection (because it will never be the top-priority issue), but I don't want to give up on forward references nor introspection.
Mar 09
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 10 March 2021 at 02:46:26 UTC, Timon Gehr wrote:
 I also think it is quite safe to assume that DMD will never 
 implement any consistent semantics for the interaction of 
 forward references and introspection (because it will never be 
 the top-priority issue), but I don't want to give up on forward 
 references nor introspection.
I hope that we can fix that. Any serious advance of meta-programming in D is basically not possible without having this issue fixed. And I think that, slowly transitioning the compiler to an asynchronous task based system (get resolution and semantic analysis to be based on co-routines); can get us there.
Mar 09
parent reply Max Haughton <maxhaton gmail.com> writes:
On Wednesday, 10 March 2021 at 06:53:37 UTC, Stefan Koch wrote:
 On Wednesday, 10 March 2021 at 02:46:26 UTC, Timon Gehr wrote:
 I also think it is quite safe to assume that DMD will never 
 implement any consistent semantics for the interaction of 
 forward references and introspection (because it will never be 
 the top-priority issue), but I don't want to give up on 
 forward references nor introspection.
I hope that we can fix that. Any serious advance of meta-programming in D is basically not possible without having this issue fixed. And I think that, slowly transitioning the compiler to an asynchronous task based system (get resolution and semantic analysis to be based on co-routines); can get us there.
Is that realistic for dmd?
Mar 10
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 10 March 2021 at 20:17:57 UTC, Max Haughton wrote:
 On Wednesday, 10 March 2021 at 06:53:37 UTC, Stefan Koch wrote:
 On Wednesday, 10 March 2021 at 02:46:26 UTC, Timon Gehr wrote:
 I also think it is quite safe to assume that DMD will never 
 implement any consistent semantics for the interaction of 
 forward references and introspection (because it will never 
 be the top-priority issue), but I don't want to give up on 
 forward references nor introspection.
I hope that we can fix that. Any serious advance of meta-programming in D is basically not possible without having this issue fixed. And I think that, slowly transitioning the compiler to an asynchronous task based system (get resolution and semantic analysis to be based on co-routines); can get us there.
Is that realistic for dmd?
Let me say it like this. I think It is more realistic than giving up on dmd.
Mar 10
next sibling parent reply Max Haughton <maxhaton gmail.com> writes:
On Wednesday, 10 March 2021 at 21:06:52 UTC, Stefan Koch wrote:
 On Wednesday, 10 March 2021 at 20:17:57 UTC, Max Haughton wrote:
 On Wednesday, 10 March 2021 at 06:53:37 UTC, Stefan Koch wrote:
 On Wednesday, 10 March 2021 at 02:46:26 UTC, Timon Gehr wrote:
 [...]
I hope that we can fix that. Any serious advance of meta-programming in D is basically not possible without having this issue fixed. And I think that, slowly transitioning the compiler to an asynchronous task based system (get resolution and semantic analysis to be based on co-routines); can get us there.
Is that realistic for dmd?
Let me say it like this. I think It is more realistic than giving up on dmd.
I agree although maybe I should rephrase - what's the gameplan? Cull mutability first, then restructure, then go to some abstraction like coroutines?
Mar 10
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 10 March 2021 at 21:39:05 UTC, Max Haughton wrote:
 On Wednesday, 10 March 2021 at 21:06:52 UTC, Stefan Koch wrote:
 On Wednesday, 10 March 2021 at 20:17:57 UTC, Max Haughton 
 wrote:
 On Wednesday, 10 March 2021 at 06:53:37 UTC, Stefan Koch 
 wrote:
 On Wednesday, 10 March 2021 at 02:46:26 UTC, Timon Gehr 
 wrote:
 [...]
I hope that we can fix that. Any serious advance of meta-programming in D is basically not possible without having this issue fixed. And I think that, slowly transitioning the compiler to an asynchronous task based system (get resolution and semantic analysis to be based on co-routines); can get us there.
Is that realistic for dmd?
Let me say it like this. I think It is more realistic than giving up on dmd.
I agree although maybe I should rephrase - what's the gameplan? Cull mutability first, then restructure, then go to some abstraction like coroutines?
I would like to get a work-stealing task/job system first. And then slowly bit by bit rewrite pieces of semantic to use it. I am not _sure_ if I can make the transition between the systems incremental, but I will at least try. that said. The very first step is to draw the graph that semantic currently walks. That alone is unfortunately a huge pain.
Mar 10
prev sibling next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Wednesday, 10 March 2021 at 21:06:52 UTC, Stefan Koch wrote:
 On Wednesday, 10 March 2021 at 20:17:57 UTC, Max Haughton wrote:
 Is that realistic for dmd?
Let me say it like this. I think It is more realistic than giving up on dmd.
I only have questions about that idea of co-routines. - would this really solve issues ? - what is the idea on how to drive the co-routines ? - launch blindly new ones when the PASS value of a node is not sufficent ? and if so, how would that give different results of what is done actually ? - dont you rather think that this should be driven by a system of graph to determine the right order ? - are there compilers for other other languages that are forward and implement this kind of "next gen" way of driving the semantic ? There need to be some kind of proof showing the benefit. Pick an example an describe what is done for now, why this does not work then describe why this would work better with fibers.
Mar 10
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 10 March 2021 at 23:29:21 UTC, Basile B. wrote:
 On Wednesday, 10 March 2021 at 21:06:52 UTC, Stefan Koch wrote:
 On Wednesday, 10 March 2021 at 20:17:57 UTC, Max Haughton 
 wrote:
 Is that realistic for dmd?
Let me say it like this. I think It is more realistic than giving up on dmd.
I only have questions about that idea of co-routines. - would this really solve issues ? - what is the idea on how to drive the co-routines ? - launch blindly new ones when the PASS value of a node is not sufficent ? and if so, how would that give different results of what is done actually ? - dont you rather think that this should be driven by a system of graph to determine the right order ? - are there compilers for other other languages that are forward and implement this kind of "next gen" way of driving the semantic ? There need to be some kind of proof showing the benefit. Pick an example an describe what is done for now, why this does not work then describe why this would work better with fibers.
Just look at sdc if you want to have an example of what's better. The main benefit of fibers is not that they magically resolve all the ordering issues, the main benefit it that they allow you to cleanly suspend the resolving process and continue at a later time when it's dependencies are resolved. The graph still needs to be explored and analyzed by the scheduler. JAI uses a similar structure. called "queued". Which essentially serve as scheduling units. The issue it would surely resolve "right away" is the one of shared state and leakey managed context/scope. Since for fibers to work all the state has to be determined by the fiber stack.
Mar 10
prev sibling parent Bruce Carneal <bcarneal gmail.com> writes:
On Wednesday, 10 March 2021 at 21:06:52 UTC, Stefan Koch wrote:
 On Wednesday, 10 March 2021 at 20:17:57 UTC, Max Haughton wrote:
 On Wednesday, 10 March 2021 at 06:53:37 UTC, Stefan Koch wrote:
 On Wednesday, 10 March 2021 at 02:46:26 UTC, Timon Gehr wrote:
 I also think it is quite safe to assume that DMD will never 
 implement any consistent semantics for the interaction of 
 forward references and introspection (because it will never 
 be the top-priority issue), but I don't want to give up on 
 forward references nor introspection.
I hope that we can fix that. Any serious advance of meta-programming in D is basically not possible without having this issue fixed. And I think that, slowly transitioning the compiler to an asynchronous task based system (get resolution and semantic analysis to be based on co-routines); can get us there.
Is that realistic for dmd?
Let me say it like this. I think It is more realistic than giving up on dmd.
A second front end effort could provide a number of benefits beyond those associated with hindsight and a clean sheet architecture. Not the least of these is the ability to compare two implementations. Whether it is worth redeploying some dmd resources to a second front end is a decision for those writing the checks (or donating their time) but here's a hypothetical that may help: if you had abundant resources at your disposal would you restrict your efforts to dmd only? I would not. Given "abundant" resources, restricting efforts to dmd sounds much too conservative. Also, to answer the next question, "abundant" for me means any amount of time that I have to dedicate to compiler work which, unfortunately, currently amounts to approximately a hill of beans (a few hours now and then exploring SDC, LLVM, MLIR, and SPIR-V). Here's hoping "abundant" resources come in to view and/or incremental improvement of dmd leads towards simplicity and robustness.
Mar 10