www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - D compilation is too slow and I am forking the compiler

reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-forking-the-compiler/
Nov 21
next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Wednesday, 21 November 2018 at 08:07:52 UTC, Vladimir 
Panteleev wrote:
 https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-forking-the-compiler/
You gave me a fright there with the title there for a moment. Awesome stuff though. Not sure how easy it will be to upstream considering this needs to not wreck Windows and needs to work with LDC/GDC (at least we have inlining in the backend).
Nov 21
parent Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 21 November 2018 at 08:32:39 UTC, Nicholas Wilson 
wrote:
 You gave me a fright there with the title there for a moment.
:)
 Awesome stuff though. Not sure how easy it will be to upstream 
 considering this needs to not wreck Windows and needs to work 
 with LDC/GDC (at least we have inlining in the backend).
All the DMD-side logic is all encapsulated in one function: https://github.com/CyberShadow/dmd/blob/dmdforker/src/dmd/mars.d#L501-L673 Its body can be versioned out in incompatible platforms/implementations.
Nov 21
prev sibling next sibling parent bauss <jj_1337 live.dk> writes:
On Wednesday, 21 November 2018 at 08:07:52 UTC, Vladimir 
Panteleev wrote:
 https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-forking-the-compiler/
Not only an interesting read, but also interesting research!
Nov 21
prev sibling next sibling parent Iain Buclaw <ibuclaw gdcproject.org> writes:
On Wednesday, 21 November 2018 at 08:07:52 UTC, Vladimir 
Panteleev wrote:
 https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-forking-the-compiler/
You might want to have a brush up on which direction C++ modules are heading in. Notable talks would be those given at the GNU Cauldron for both 2017 and 2018. The general run-down as I understand it. === Problem to solve: Compiler asks an Oracle about module A. Phrased this way, Compiler is a client, Oracle is a server. Oracle could be a file, socket, remote server, anything that can be read from or written to. Communication can be done via a standard format (such as json). This means that the Oracle (the implementation of) that keeps track of compilation and dependencies of the build is now someone else's problem as far as the Compiler is concerned. === I think what you've already started would fit well into this. Iain.
Nov 21
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/21/2018 12:07 AM, Vladimir Panteleev wrote:
 https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-f
rking-the-compiler/ 
I implemented precompiled headers for Digital Mars C++. It took a loooong time to work all the bugs out of it. It's also a brittle system. It works by allocating memory from a memory-mapped file, which serves as the precompiled header.
Nov 21
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 21 November 2018 at 09:46:44 UTC, Walter Bright 
wrote:
 It works by allocating memory from a memory-mapped file, which 
 serves as the precompiled header.
Hey, that's a great idea! Can we do this for DMD? :D On a more serious note: do you think that with D's features (type system / metaprogramming), you could have avoided some of those bugs? For example, one thing we can do in D which is still impossible in C++ is to automatically serialize/deserialize all fields of a struct/class (using tupleof / allMembers).
Nov 21
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/21/2018 2:16 AM, Vladimir Panteleev wrote:
 On Wednesday, 21 November 2018 at 09:46:44 UTC, Walter Bright wrote:
 It works by allocating memory from a memory-mapped file, which serves as the 
 precompiled header.
Hey, that's a great idea! Can we do this for DMD? :D On a more serious note: do you think that with D's features (type system / metaprogramming), you could have avoided some of those bugs? For example, one thing we can do in D which is still impossible in C++ is to automatically serialize/deserialize all fields of a struct/class (using tupleof / allMembers).
Memory mapped files really were the key to success, because if you could reload the mmf at the same address, the pointers did not have to be patched. In the DMC++ source code, "dehydrating" a pointer meant subtracting a value from it so it was correct for the base address of the mmf, and "hydrating" a pointer was the inverse. The two bug prone problems were: 1. separating out the tangled data structures into what goes into the pch, and what does not. Obviously, nothing in the pch could point outside of it. 2. .h files are simply not compatible with this, so you've got to detect when it won't work. For example, anything like a command line switch or a macro that might cause different code to be generated in the pch had to invalidate it. Maybe I should have done your fork idea? :-) My experience with this drove many design decisions for D modules, for example, D modules are unaffected by where they are imported, the order they are imported, or the number of times they are imported. (Yes, I know about https://digitalmars.com/d/archives/digitalmars/D/D_needs_to_be_honest_320976.html) Anyhow, what I've thought about doing since the beginning was make DMD multithreaded. The language is designed to support multithreaded compilation. For example, lexing, parsing, semantic analysis, optimization, and code generation can all be done concurrently. DMD 1.0 would read imports in a separate thread. This would speed things up if you were using a slow filesystem, like NAS or a USB stick, but it was eventually disabled because there wasn't a perceptible speedup with current filesystems. Wouldn't it be awesome to have the lexing/parsing of the imports all done in parallel? The main difficulty in getting that to work is dealing with the shared string table.
Nov 21
next sibling parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On Wednesday, 21 November 2018 at 10:56:02 UTC, Walter Bright 
wrote:
 Wouldn't it be awesome to have the lexing/parsing of the 
 imports all done in parallel? The main difficulty in getting 
 that to work is dealing with the shared string table.
What about creating a new Fiber for each module needing lexing/parsing/semantic to be ran? Compilation of one module would then get as far as it can until it needs to defer, then calls yield() to continue compilation of the next module. This in hope that when the round trip returns, the AST will be sufficiently complete enough for compilation to continue.
Nov 21
parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/21/2018 3:19 AM, Iain Buclaw wrote:
 On Wednesday, 21 November 2018 at 10:56:02 UTC, Walter Bright wrote:
 Wouldn't it be awesome to have the lexing/parsing of the imports all done in 
 parallel? The main difficulty in getting that to work is dealing with the 
 shared string table.
What about creating a new Fiber for each module needing lexing/parsing/semantic to be ran?  Compilation of one module would then get as far as it can until it needs to defer, then calls yield() to continue compilation of the next module. This in hope that when the round trip returns, the AST will be sufficiently complete enough for compilation to continue.
Since the lexing/parsing does not do any blocking calls, fibers won't help. It has to be another hardware thread. Trying to parallelize semantic over multiple modules will be very hard to do.
Nov 21
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2018-11-21 11:56, Walter Bright wrote:

 Wouldn't it be awesome to have the lexing/parsing of the imports all 
 done in parallel? The main difficulty in getting that to work is dealing 
 with the shared string table.
Would it be possible to have one string table per thread and merge them to one single shared string table before continuing with the next phase? -- /Jacob Carlborg
Nov 23
parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/23/2018 2:12 AM, Jacob Carlborg wrote:
 Would it be possible to have one string table per thread and merge them to one 
 single shared string table before continuing with the next phase?
It'd probably be even slower because one would have to rewrite all the pointers into the string table.
Nov 23
prev sibling parent reply welkam <wwwelkam gmail.com> writes:
On Wednesday, 21 November 2018 at 10:56:02 UTC, Walter Bright 
wrote:
 Wouldn't it be awesome to have the lexing/parsing of the 
 imports all done in parallel?
From my testing lexing/parsing takes small amount of build time so running it in parallel might be small gain. We should consider running in parallel more heavy hitting features like CTFE and templates. Since we are in wish land here is my wishes. Currently D reads the all files that are passed in command line before starting lexing/parsing, but in principle we could start lexing/parsing after first file is read. In fact we could start after first file`s first line is read. Out of all operation before semantic pass, reading from hard disk should be the slowest so it might be possible to decode utf-8, lex and parse at the speed of reading from hard disk. If we run these steps in different thread on the same core with SMT we could better use core`s resources. Reading file with kernel, decoding UTF-8 with vector instructions and lexing/parsing with scalar operations while all communication is done trough L1 and L2 cache. I thought about using memory mapped files to unblock file reading as a first step but lack of good documentation about mmf and lack of thorough understanding of front end made me postpone this modification. Its a change with little benefit.
 The main difficulty in getting that to work is dealing with the 
 shared string table.
At begging of parsing a thread could get a read-only shared slice of string table. All strings not in table are put in local string table. After parsing tables are merged and shared slice is updated so new thread could start with bigger table. this assumes that table is not sorted
Nov 23
next sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Friday, 23 November 2018 at 13:23:22 UTC, welkam wrote:
 If we run these steps in different thread on the same core with 
 SMT we could better use core`s resources. Reading file with 
 kernel, decoding UTF-8 with vector instructions and 
 lexing/parsing with scalar operations while all communication 
 is done trough L1 and L2 cache.
You might save some pages from the data cache, but by doing more work at once, the code might stop fitting in the execution-related caches (code pages, microcode, branch prediction) instead.
Nov 23
parent welkam <wwwelkam gmail.com> writes:
On Friday, 23 November 2018 at 14:32:39 UTC, Vladimir Panteleev 
wrote:
 On Friday, 23 November 2018 at 13:23:22 UTC, welkam wrote:
 If we run these steps in different thread on the same core 
 with SMT we could better use core`s resources. Reading file 
 with kernel, decoding UTF-8 with vector instructions and 
 lexing/parsing with scalar operations while all communication 
 is done trough L1 and L2 cache.
You might save some pages from the data cache, but by doing more work at once, the code might stop fitting in the execution-related caches (code pages, microcode, branch prediction) instead.
Its not about saving tlb pages or fitting better in cache. Compilers are considered streaming applications - they dont utilize cpu caches effectively. You cant read one character and emit machine code then read next character you have to go over all data multiple times while you modify it. I can find white papers if you interested where people test GCC with different cache architectures and it doesnt make much of a difference. GCC is popular application when testing caches. Here are profiling data from DMD Performance counter stats for 'dmd -c main.d': 600.77 msec task-clock:u # 0.803 CPUs utilized 0 context-switches:u # 0.000 K/sec 0 cpu-migrations:u # 0.000 K/sec 33,209 page-faults:u # 55348.333 M/sec 1,072,289,307 cycles:u # 1787148.845 GHz 870,175,210 stalled-cycles-frontend:u # 81.15% frontend cycles idle 721,897,927 stalled-cycles-backend:u # 67.32% backend cycles idle 881,895,208 instructions:u # 0.82 insn per cycle # 0.99 stalled cycles per insn 171,211,752 branches:u # 285352920.000 M/sec 11,287,327 branch-misses:u # 6.59% of all branches 0.747720395 seconds time elapsed 0.497698000 seconds user 0.104165000 seconds sys Most important data in this conversation is 0.82 insn per cycle. My CPU could do ~2 IPC so there are plenty of CPU resources available. New Intel desktop processors are designed to do 4 insn/cycle. What is limiting DMD performance is slow RAM, data fetching and not what you listed. code pages - you mean TLB here? microcode cache. Not all processors have it and those who have only benefit trivial loops. DMD have complex loops. branch prediction. More entries in branch predictor wont help here because branches are missed because data is unpredictable not because there are too many branches. Also branch missprediction penalty is around 30 cycles while reading from RAM could be over 200 cycles. L1 code cache. You didnt mention this but running those tasks in SMT mode might trash L1$ so execution might not be optimal. Instead of parallel reading of imports DMD needs more data oriented data structures instead of old OOP inspired data structures. Ill give you example why its the case. Consider struct { bool isAlive; <other data at least 7 bytes of size> } If you want to read data from that bool CPU needs to fetch 8 bytes of data(cache line of 64 bits). What this means is that for one bit of information CPU fetches 64 bits of data resulting in 1/64 = 0.015625 or ~1.6 % signal to noise ratio. This is terrible! AFAIK DMD doesnt make this kind of mistake but its full of large structs and classes that are not efficient to read. To fix this we need to split those large data structures into smaller ones that only contain what is needed for particular algorithm. I predict 2x speed improvement if we transform all data structures in DMD. Thats improvement without improving algorithms only changing data structures. This getting too longs so i will stop right now
Nov 23
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/23/2018 5:23 AM, welkam wrote:
 Currently D reads the all files 
 that are passed in command line before starting lexing/parsing, but in
principle 
 we could start lexing/parsing after first file is read. In fact we could start 
 after first file`s first line is read.
DMD used to do that. But it was removed because: 1. nobody understood the logic 2. it didn't seem to make a difference You can still see the vestiges by the: static if (ASYNCREAD) blocks in the code.
Nov 23
parent welkam <wwwelkam gmail.com> writes:
On Friday, 23 November 2018 at 19:21:03 UTC, Walter Bright wrote:
 On 11/23/2018 5:23 AM, welkam wrote:
 Currently D reads the all files that are passed in command 
 line before starting lexing/parsing, but in principle we could 
 start lexing/parsing after first file is read. In fact we 
 could start after first file`s first line is read.
DMD used to do that. But it was removed because: 1. nobody understood the logic 2. it didn't seem to make a difference You can still see the vestiges by the: static if (ASYNCREAD) blocks in the code.
I didnt expect huge wins. This would be useful when you start your computer and files have to be read from old spinning rust and the project has many files. Otherwise files will be cached and memcopy is fast. I was surprised on how fast modern computers copy data from one place to another. Speaking of memcpy here is a video you might like. It has memcpy, assembler and a bit of compiler. Its very easy watch for when you want to relax. Level1 Diagnostic: Fixing our Memcpy Troubles (for Looking Glass) https://www.youtube.com/watch?v=idauoNVwWYE
Nov 23
prev sibling next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Wednesday, 21 November 2018 at 08:07:52 UTC, Vladimir 
Panteleev wrote:
 https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-forking-the-compiler/
This is #2 on HN at the moment.
Nov 21
next sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Wednesday, 21 November 2018 at 11:18:22 UTC, Nicholas Wilson 
wrote:
 On Wednesday, 21 November 2018 at 08:07:52 UTC, Vladimir 
 Panteleev wrote:
 https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-forking-the-compiler/
This is #2 on HN at the moment.
I would be wary of such titles. Whatever will remain in minds will be something like "D compilation is slow" which isn't accurate when compared to the competition. The article is clever, the title is clever, but most people will only read the title.
Nov 21
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/21/2018 5:55 AM, Guillaume Piolat wrote:
 On Wednesday, 21 November 2018 at 11:18:22 UTC, Nicholas Wilson wrote:
 On Wednesday, 21 November 2018 at 08:07:52 UTC, Vladimir Panteleev wrote:
 https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-f
rking-the-compiler/ 
This is #2 on HN at the moment.
I would be wary of such titles. Whatever will remain in minds will be something like "D compilation is slow" which isn't accurate when compared to the competition. The article is clever, the title is clever, but most people will only read the title.
Unfortunately, you're right. The title will leave the impression "D is slow at compiling". You have to carefully read the article to see otherwise, and few will do that.
Nov 21
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
I neglected point out, however, that the article itself is a home run! Thank 
you, Vladimir!
Nov 21
prev sibling next sibling parent Daniel Kozak <kozzi11 gmail.com> writes:
I would say opposite :)

On Wed, Nov 21, 2018 at 9:55 PM Walter Bright via Digitalmars-d-announce <
digitalmars-d-announce puremagic.com> wrote:

 On 11/21/2018 5:55 AM, Guillaume Piolat wrote:
 On Wednesday, 21 November 2018 at 11:18:22 UTC, Nicholas Wilson wrote:
 On Wednesday, 21 November 2018 at 08:07:52 UTC, Vladimir Panteleev
wrote:

https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-forking-the-compiler/

 This is #2 on HN at the moment.
I would be wary of such titles. Whatever will remain in minds will be something like "D compilation is
slow"
 which isn't accurate when compared to the competition.

 The article is clever, the title is clever, but most people will only
read the
 title.
Unfortunately, you're right. The title will leave the impression "D is slow at compiling". You have to carefully read the article to see otherwise, and few will do that.
Nov 21
prev sibling next sibling parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 21 November 2018 at 20:51:17 UTC, Walter Bright 
wrote:
 Unfortunately, you're right. The title will leave the 
 impression "D is slow at compiling". You have to carefully read 
 the article to see otherwise, and few will do that.
Sorry about that. I'll have to think of two titles next time, one for the D community and one for everyone else. If it's of any consolation, the top comments in both discussion threads point out that the title is inaccurate on purpose.
Nov 21
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/21/2018 8:48 PM, Vladimir Panteleev wrote:
 On Wednesday, 21 November 2018 at 20:51:17 UTC, Walter Bright wrote:
 Unfortunately, you're right. The title will leave the impression "D is slow at 
 compiling". You have to carefully read the article to see otherwise, and few 
 will do that.
Sorry about that. I'll have to think of two titles next time, one for the D community and one for everyone else. If it's of any consolation, the top comments in both discussion threads point out that the title is inaccurate on purpose.
It has indeed done well. In fact, the article is so good it is probably worth it to have that attention-getting title. It is a risky strategy, though :-)
Nov 21
prev sibling next sibling parent reply welkam <wwwelkam gmail.com> writes:
On Thursday, 22 November 2018 at 04:48:09 UTC, Vladimir Panteleev 
wrote:
 Sorry about that. I'll have to think of two titles next time, 
 one for the D community and one for everyone else.

 If it's of any consolation, the top comments in both discussion 
 threads point out that the title is inaccurate on purpose.
Your post on reddit received more comments than D front ends inclusion to GCC. If you titled your post differently you probably wouldn't had such success so from my perspective its a net positive. Sure there are few people that took the wrong message but there are more people who saw your post
Nov 23
parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/23/2018 6:37 AM, welkam wrote:
 Your post on reddit received more comments than D front ends inclusion to GCC. 
 If you titled your post differently you probably wouldn't had such success so 
 from my perspective its a net positive. Sure there are few people that took
the 
 wrong message but there are more people who saw your post
It definitely shows the value of a provocative title!
Nov 23
prev sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Thursday, 22 November 2018 at 04:48:09 UTC, Vladimir Panteleev 
wrote:
 On Wednesday, 21 November 2018 at 20:51:17 UTC, Walter Bright 
 wrote:
 Unfortunately, you're right. The title will leave the 
 impression "D is slow at compiling". You have to carefully 
 read the article to see otherwise, and few will do that.
Sorry about that. I'll have to think of two titles next time, one for the D community and one for everyone else. If it's of any consolation, the top comments in both discussion threads point out that the title is inaccurate on purpose.
Please don't get me wrong, it's an excellent article, a provocative title, and fantastic work going on. I didn't meant to hurt! In my opinion language adoption is a seduction/sales process very much like business-to-consumer is, the way I see it it's strikingly similar to marketing B2C apps, unless there will be no "impulse buy". Actually no less than 3 programmer friends came to (I'm the weirdo-using-D and people are _always_ in disbelief and invent all sorts of reasons not to try) saying they saw an article on D on HN, with "D compilation is slow", and on further examination they didn't read or at best the first paragraph. But they did remember the title. They may rationally think their opinion of D hasn't changed: aren't we highly capable people? I'm not making that up! So why is it a problem ? HN may be the only time they hear about D. The words of the title may be their only contact with it. The first 3 words of the title may be the only thing associated with the "D language" chunk in their brain. The associative mind doesn't know _negation_ so even a title like "D compilation wasn't fast so I forked the compiler" is better from a marketing point of view since it contains the word "fast" in it! That's why marketing people have the annoying habit of using positive words, you may think this stuff is unimportant but this is actually the important meat. Reasonable people may think marketing and biases don't apply to them but they do, it works without your consent.
Nov 26
next sibling parent reply Joakim <dlang joakim.fea.st> writes:
On Monday, 26 November 2018 at 16:00:36 UTC, Guillaume Piolat 
wrote:
 On Thursday, 22 November 2018 at 04:48:09 UTC, Vladimir 
 Panteleev wrote:
 On Wednesday, 21 November 2018 at 20:51:17 UTC, Walter Bright 
 wrote:
 Unfortunately, you're right. The title will leave the 
 impression "D is slow at compiling". You have to carefully 
 read the article to see otherwise, and few will do that.
Sorry about that. I'll have to think of two titles next time, one for the D community and one for everyone else. If it's of any consolation, the top comments in both discussion threads point out that the title is inaccurate on purpose.
Please don't get me wrong, it's an excellent article, a provocative title, and fantastic work going on. I didn't meant to hurt! In my opinion language adoption is a seduction/sales process very much like business-to-consumer is, the way I see it it's strikingly similar to marketing B2C apps, unless there will be no "impulse buy".
I find that hard to believe: we are talking about a technical tool here. Also, regardless of how languages are chosen as they get into the majority, D is very much still in the innovators/early-adopters stage: https://en.m.wikipedia.org/wiki/Technology_adoption_life_cycle That is a very different type of sales process, much more geared towards what the new tech can actually do.
 Actually no less than 3 programmer friends came to (I'm the 
 weirdo-using-D and people are _always_ in disbelief and invent 
 all sorts of reasons not to try) saying they saw an article on 
 D on HN, with "D compilation is slow", and on further 
 examination they didn't read or at best the first paragraph. 
 But they did remember the title. They may rationally think 
 their opinion of D hasn't changed: aren't we highly capable 
 people?
With people like that, it's almost impossible to get them in the early adopter stage. They will only jump on the bandwagon once it's full, ie as part of the late majority.
 I'm not making that up! So why is it a problem ?

 HN may be the only time they hear about D. The words of the 
 title may be their only contact with it. The first 3 words of 
 the title may be the only thing associated with the "D 
 language" chunk in their brain.

 The associative mind doesn't know _negation_ so even a title 
 like "D compilation wasn't fast so I forked the compiler" is 
 better from a marketing point of view since it contains the 
 word "fast" in it! That's why marketing people have the 
 annoying habit of using positive words, you may think this 
 stuff is unimportant but this is actually the important meat.

 Reasonable people may think marketing and biases don't apply to 
 them but they do, it works without your consent.
I agree that it was a risky title, as many who don't know D will simply see it and go, "Yet another slow compiler, eh, I'll pass" and not click on the link. Whereas others who have heard something of D will be intrigued, as they know it's already supposed to compile fast. And yet more others will click on it purely for the controversy, just to gawk at some technical bickering. Given how well it did on HN/reddit/lobste.rs, I think Vlad's gamble probably paid off. We can't run the counterfactual of choosing a safer title to see if it would have done even better, let's just say it did well enough. ;)
Nov 26
next sibling parent reply bachmeier <no spam.net> writes:
On Monday, 26 November 2018 at 16:21:39 UTC, Joakim wrote:

 I agree that it was a risky title, as many who don't know D 
 will simply see it and go, "Yet another slow compiler, eh, I'll 
 pass" and not click on the link. Whereas others who have heard 
 something of D will be intrigued, as they know it's already 
 supposed to compile fast. And yet more others will click on it 
 purely for the controversy, just to gawk at some technical 
 bickering.
I don't actually think it was risky. What are the odds that someone was going to start using D for a major project but then changed her mind upon seeing a title on HN or Reddit? Probably very small that even one person did that. On the other hand, it says a lot of other things: - There's an active community that cares about the language. - It's not a dying language. - Fast compilation is a realistic possibility. - There are users with the technical ability to make the compiler faster. And then there is always the fact that there was a story on HN/Reddit about D. It's hard for publicity for a language like D to be bad when so few people use it.
Nov 26
parent reply Joakim <dlang joakim.fea.st> writes:
On Monday, 26 November 2018 at 16:42:40 UTC, bachmeier wrote:
 On Monday, 26 November 2018 at 16:21:39 UTC, Joakim wrote:

 I agree that it was a risky title, as many who don't know D 
 will simply see it and go, "Yet another slow compiler, eh, 
 I'll pass" and not click on the link. Whereas others who have 
 heard something of D will be intrigued, as they know it's 
 already supposed to compile fast. And yet more others will 
 click on it purely for the controversy, just to gawk at some 
 technical bickering.
I don't actually think it was risky. What are the odds that someone was going to start using D for a major project but then changed her mind upon seeing a title on HN or Reddit? Probably very small that even one person did that.
Yes, but you're ignoring the much larger group I mentioned- those who only vaguely heard of D, if at all- and the negative title gives them a reason not to look into it further.
 And then there is always the fact that there was a story on 
 HN/Reddit about D. It's hard for publicity for a language like 
 D to be bad when so few people use it.
The quote that "there's no such thing as bad publicity" comes from art and show business though, don't think it's true for tech and other markets. When your audience is looking for a tool and not entertainment, there's lots of ways for bad publicity to sink it. Anyway, I noted in this case that the provocative title may actually have gotten more people to read a positive post, so the pros likely outweighed the cons. We can just never know how large the unclicked-on downside was: you can never measure how many people heard of but _didn't_ buy your book, because they didn't like the title or something else about its exterior. On Monday, 26 November 2018 at 16:53:59 UTC, Guillaume Piolat wrote:
 On Monday, 26 November 2018 at 16:21:39 UTC, Joakim wrote:
 In my opinion language adoption is a seduction/sales process 
 very much like business-to-consumer is, the way I see it it's 
 strikingly similar to marketing B2C apps, unless there will 
 be no "impulse buy".
I find that hard to believe: we are talking about a technical tool here.
How many times have you been in this conversation: -------------------------- - What language are you using? - D. - I know next to nothing about D. - Oh, it's very good, I even built a business on it! <laundry list of arguments and features>. - Oh no thanks. I should try Rust, it's secure, fast, modern blah blah; facts don't matter to me. But in reality I won't even learn a new language, I'm happy with a language without multi-threading. -------------------------- It happens to me ALL THE TIME. This pattern is so predictable it's becoming boring so now I just keep silent.
Never, I don't go around trying to convince people one-on-one to use D. I have given talks to groups introducing the language, that's how I go about it.
 What happens? Rust / Go have outmarketed us with words.

 The battle (of marketing) is on words not technical features, 
 Rust happen to own "programming language" + "safety", what do 
 we own? D is good in all kinds of directions and the marketing 
 message is less simple.

 The leaders choose to own the word "fast" (see our new motto 
 "fast code, fast" which is very accurate) and it's important to 
 get aligned.
I'll note that in your example they haven't actually learnt Rust either. I don't think marketing is that relevant for D at this stage, nor for Rust/Go either. The way anything- tech, fashion, TV shows- becomes popular is that some early tastemaker decides that it's worth using or backing. Eventually, enough early adopters find value that it spreads out to the majority, who simply follow their lead. Most people aren't early adopters of most things. They like to think they are, so they'll give you all kinds of rational-sounding reasons for why they don't like some new tech, but the real underlying thought process goes something like this, "I have no idea if this new tech will do well or not. I could take a risk on it but it's safer not to, so I will just wait and see if it gets popular, then follow the herd." Very few will admit this though, hence the list of plausible-sounding reasons that don't actually make sense! ;) As Laeeth always says, you're best off looking for people who're actually capable and empowered to make such risky decisions, rather than aiming for the majority too early, because they only jump on board once the bandwagon is stuffed and rolling downhill.
 Also, regardless of how languages are chosen as they get into 
 the majority, D is very much still in the 
 innovators/early-adopters stage:
But the current state of D would very much accomodate the middle-of-the-curve adopters. The language rarely breaks stuff. People making money with it, making long-term bets. Hell, I could make a laundry list of things that are better in D versus any alternatives! That doesn't bring users.
I'm not talking about the quality of the product. I'm talking about the current size of the userbase, which is still in the early adopter stage.
 With people like that, it's almost impossible to get them in 
 the early adopter stage. They will only jump on the bandwagon 
 once it's full, ie as part of the late majority.
There is a gap where we are, but "People like that" are almost everyone.
Yes, this is why you must ignore almost everyone. It is a waste of time, because they will not take a risk with new tech.
 Those people actually are middle-of-the-curve adopter, if you 
 see a true late adopter in the wild it takes 3 relatives 
 programming in D so that they start to be interested.
Heh, perhaps.
 Who doesn't want to be out of the early adopter stage, and get 
 into the "officially endorsed safe choice" cohort?
All of us, but you cannot skip steps, not how it works.
 D is remarkably ready as a safe choice for lots of software.


 Given how well it did on HN/reddit/lobste.rs, I think Vlad's 
 gamble probably paid off. We can't run the counterfactual of 
 choosing a safer title to see if it would have done even 
 better, let's just say it did well enough. ;)
Alternative darker view: ever remarked how D articles often goes downvoted on HN? The title who says something bad about D is upvoted ; it's easy to see events as going our way. I, for one, didn't really read the article. Who has time for that?
Maybe you should read it, some of the proggitors even created a new reddit sub based on this post, coining the term ClickGold, for when a clickbait article ends up being much _more_ interesting than its leading title: ;) https://www.reddit.com/r/programming/comments/9z36xg/comment/ea6afce
Nov 27
parent Guillaume Piolat <first.last gmail.com> writes:
On Tuesday, 27 November 2018 at 14:19:12 UTC, Joakim wrote:
 As Laeeth always says, you're best off looking for people 
 who're actually capable and empowered to make such risky 
 decisions, rather than aiming for the majority too early, 
 because they only jump on board once the bandwagon is stuffed 
 and rolling downhill.
I get the (I read "D compilation is slow" + "I'm not interested") vibe from people that actually _are_ early adopters, very interested in programming, who adopted Scala and Nim despite obvious risks. I disagree with your opinion but just lacks the time to defend my own in a lengthy forum post. Let me make a giant argument by authority: I sell B2C software for a living. Convincing people to try software despite their will is what I do for a living. "Build it and they will come" would be a fantasy to believe if we hadn't competitors who had people come way before they built it. Complicated triple negation arguments (D compile fast! no wait, it compiles slowly! no wait, it compiles fast!) don't work. In a few months whenever someone bring out D, forum replies will say "but D compilation is not that fast".
Nov 27
prev sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Monday, 26 November 2018 at 16:21:39 UTC, Joakim wrote:
 In my opinion language adoption is a seduction/sales process 
 very much like business-to-consumer is, the way I see it it's 
 strikingly similar to marketing B2C apps, unless there will be 
 no "impulse buy".
I find that hard to believe: we are talking about a technical tool here.
How many times have you been in this conversation: -------------------------- - What language are you using? - D. - I know next to nothing about D. - Oh, it's very good, I even built a business on it! <laundry list of arguments and features>. - Oh no thanks. I should try Rust, it's secure, fast, modern blah blah; facts don't matter to me. But in reality I won't even learn a new language, I'm happy with a language without multi-threading. -------------------------- It happens to me ALL THE TIME. This pattern is so predictable it's becoming boring so now I just keep silent. What happens? Rust / Go have outmarketed us with words. The battle (of marketing) is on words not technical features, Rust happen to own "programming language" + "safety", what do we own? D is good in all kinds of directions and the marketing message is less simple. The leaders choose to own the word "fast" (see our new motto "fast code, fast" which is very accurate) and it's important to get aligned.
 Also, regardless of how languages are chosen as they get into 
 the majority, D is very much still in the 
 innovators/early-adopters stage:
But the current state of D would very much accomodate the middle-of-the-curve adopters. The language rarely breaks stuff. People making money with it, making long-term bets. Hell, I could make a laundry list of things that are better in D versus any alternatives! That doesn't bring users.
 With people like that, it's almost impossible to get them in 
 the early adopter stage. They will only jump on the bandwagon 
 once it's full, ie as part of the late majority.
There is a gap where we are, but "People like that" are almost everyone. Those people actually are middle-of-the-curve adopter, if you see a true late adopter in the wild it takes 3 relatives programming in D so that they start to be interested. Who doesn't want to be out of the early adopter stage, and get into the "officially endorsed safe choice" cohort? D is remarkably ready as a safe choice for lots of software.
 Given how well it did on HN/reddit/lobste.rs, I think Vlad's 
 gamble probably paid off. We can't run the counterfactual of 
 choosing a safer title to see if it would have done even 
 better, let's just say it did well enough. ;)
Alternative darker view: ever remarked how D articles often goes downvoted on HN? The title who says something bad about D is upvoted ; it's easy to see events as going our way. I, for one, didn't really read the article. Who has time for that?
Nov 26
parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 11/26/18 11:53 AM, Guillaume Piolat wrote:
 
 How many times have you been in this conversation:
 
 --------------------------
 
 - What language are you using?
 - D.
 - I know next to nothing about D.
 - Oh, it's very good, I even built a business on it! <laundry list of 
 arguments and features>.
 - Oh no thanks. I should try Rust, it's secure, fast, modern blah blah; 
 facts don't matter to me. But in reality I won't even learn a new 
 language, I'm happy with a language without multi-threading.
 
 --------------------------
 
 It happens to me ALL THE TIME.
 This pattern is so predictable it's becoming boring so now I just keep 
 silent.
 
That's no surprise. The modern tech world is absolutely flooded with imbeciles and fashion-driven people.
Nov 27
prev sibling parent reply Laeeth Isharc <laeeth laeeth.com> writes:
On Monday, 26 November 2018 at 16:00:36 UTC, Guillaume Piolat 
wrote:
 On Thursday, 22 November 2018 at 04:48:09 UTC, Vladimir 
 Panteleev wrote:
 On Wednesday, 21 November 2018 at 20:51:17 UTC, Walter Bright 
 wrote:
 Unfortunately, you're right. The title will leave the 
 impression "D is slow at compiling". You have to carefully 
 read the article to see otherwise, and few will do that.
Sorry about that. I'll have to think of two titles next time, one for the D community and one for everyone else. If it's of any consolation, the top comments in both discussion threads point out that the title is inaccurate on purpose.
Please don't get me wrong, it's an excellent article, a provocative title, and fantastic work going on. I didn't meant to hurt! In my opinion language adoption is a seduction/sales process very much like business-to-consumer is, the way I see it it's strikingly similar to marketing B2C apps, unless there will be no "impulse buy".
I think that there are different strategies - decent appeal to a broad market and having a very high appeal to a small market (but there has better be something good about your potential customer base ie 'D, if you find VBA too difficult' is probably not a good strategy!). And you probably don't get to pick which situation you are in, and then one had better realise it and play the game you're in. The particular kind of market will shape what works - in my business you approach a retail client base differently from regular institutional investors and then the worlds' largest pools of money involved something else again. D isn't really marketed and it's definitely not sold. That's an implicit strategy in itself. Nassim Taleb raises the question of how do you choose between two surgeons, both recommended. One looks the part and hangs his many certificates on his office wall. The other looks scruffy with the appearance of a tradesman. Who do you pick? Taleb says pick the guy who doesn't look the part because if he got there without signalling he must have something going for him. But in general you can appeal on merits mostly to an audience that is highly discerning and very capable. If you haven't got any money to appeal to an audience that judges based on heuristics and social factors well then you can try to avoid accidentally putting people off, you can be creative with guerilla marketing but the key thing is to make the most of what you got. If everyone else does things a certain way then if for some reason that's closed off to you for now then if you look closely, with active perception,you may well see opportunities that are neglected to approach the problem another way.
 Actually no less than 3 programmer friends came to (I'm the 
 weirdo-using-D and people are _always_ in disbelief and invent 
 all sorts of reasons not to try) saying they saw an article on 
 D on HN, with "D compilation is slow", and on further 
 examination they didn't read or at best the first paragraph. 
 But they did remember the title. They may rationally think 
 their opinion of D hasn't changed: aren't we highly capable 
 people?
It doesn't matter what most people think. It matters what people who are on the fence or using D already a bit think. Or people who have a lot of problems to which D is in part a solution only they didn't know about or think of D yet. The messenger matters too. If someone you trust and rate highly tells you something based on their experience that counts for a lot more than all the blog posts in the world. And working code and lived experience dominates the social talk about it. I've talked about D with the CTO of Bloomberg, the outgoing COO of Barclays investment bank, the number two guy at a 30bn hedge fund, the COO of the largest hedge fund in the world (depending on how you count) and more. That's not going to change anything tomorrow but in time those kinds of conversations matter much more than what people might say on Reddit. It's not either /or of course, but it's just not worth sweating your reviews. Finally the reasons people buy things are not what you might reasonably think! Ask Walter how he was able to compete successfully for so long as a one man band with Microsoft. I don't think his edge was in the beginning something calculated.
 Reasonable people may think marketing and biases don't apply to 
 them but they do, it works without your consent.
The thing is that we had a bubble in synthetic manufactured marketing. And now increasingly people are tired of that and seek what's authentic, real and that doesn't pretend to be perfect. That doesn't mean a bit of thought is a bad idea,just that it might matter less than you think that the D community isn't particularly interested in marketing. Sometimes one can see that hidden in what superficially seems to be a weakness is a strength.
Nov 28
next sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Wednesday, 28 November 2018 at 12:48:46 UTC, Laeeth Isharc 
wrote:
 D isn't really marketed and it's definitely not sold.  That's 
 an implicit strategy in itself.
What I see in my (absurdly competitive) market is that the people that truly do no-marketing tend to close shop, sometimes despite very competitive offerings. It colors my perception of course, since it can be very tempting to appeal to a limited pool of discerning customers; but that would mean death. Imho the ones that succeed doing "no marketing" are actively telling others they don't do marketing. It can be sometimes funny, when the "no marketing" products comes with walls of text and videos that very much sound like... storytelling. But Like in the "no-makeup makeup", there is still makeup but not obvious. 2018's marketing trend is to subtly fake complete honesty.
Nov 28
parent Laeeth isharc <laeeth laeeth.com> writes:
On Wednesday, 28 November 2018 at 13:05:34 UTC, Guillaume Piolat 
wrote:
 On Wednesday, 28 November 2018 at 12:48:46 UTC, Laeeth Isharc 
 wrote:
 D isn't really marketed and it's definitely not sold.  That's 
 an implicit strategy in itself.
What I see in my (absurdly competitive) market is that the people that truly do no-marketing tend to close shop, sometimes despite very competitive offerings. It colors my perception of course, since it can be very tempting to appeal to a limited pool of discerning customers; but that would mean death.
What is the ratio of expenditure of your best customer to an average customer? Not much. That's one main reason why your intuition developed by organising your emotions according to your business domain fits this domain less. What is the ratio of expenditure of the biggest 'customer' of Python to the average 'customer'? Measured by resources lent to the community directly or indirectly, or by the wage bill of programmers at that firm working in Python this ratio is enormous.
Nov 29
prev sibling next sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Wednesday, 28 November 2018 at 12:48:46 UTC, Laeeth Isharc 
wrote:
 Nassim Taleb raises the question of how do you choose between 
 two surgeons, both recommended.  One looks the part and hangs 
 his many certificates on his office wall.  The other looks 
 scruffy with the appearance of a tradesman.  Who do you pick?  
 Taleb says pick the guy who doesn't look the part because if he 
 got there without signalling he must have something going for 
 him.
It's definately the kind of surgeon one should choose - programmers that are not necessarily well groomed etc.. - but is it the kind of surgeon people will actually recommend? I'm doubtful. If X has the social signalling then people will recommend X even without trying, because it's socially safe. If one doesn't have the signalling, I've found the hard way even supporters will hesitate a bit before making recommendations, because of the social standing _cost_ it may have. But then, perhaps recommendations don't matter, because opinions don't matter much? I think they matter to be even heard on public places. And I think early adopters need a nudge, the influent need to be bothered by less influents (influencers are not especially on the lookout for new options, as they are already influent). Above all I think the niche of early-adopters is smaller than the larger market for languages, and the early-adopters are going elsewhere.
Nov 28
parent reply Laeeth isharc <laeeth laeeth.com> writes:
On Wednesday, 28 November 2018 at 13:30:37 UTC, Guillaume Piolat 
wrote:
 On Wednesday, 28 November 2018 at 12:48:46 UTC, Laeeth Isharc 
 wrote:
 Nassim Taleb raises the question of how do you choose between 
 two surgeons, both recommended.  One looks the part and hangs 
 his many certificates on his office wall.  The other looks 
 scruffy with the appearance of a tradesman.  Who do you pick?  
 Taleb says pick the guy who doesn't look the part because if 
 he got there without signalling he must have something going 
 for him.
It's definately the kind of surgeon one should choose - programmers that are not necessarily well groomed etc.. - but is it the kind of surgeon people will actually recommend? I'm doubtful. If X has the social signalling then people will recommend X even without trying, because it's socially safe. If one doesn't have the signalling, I've found the hard way even supporters will hesitate a bit before making recommendations, because of the social standing _cost_ it may have. But then, perhaps recommendations don't matter, because opinions don't matter much? I think they matter to be even heard on public places. And I think early adopters need a nudge, the influent need to be bothered by less influents (influencers are not especially on the lookout for new options, as they are already influent). Above all I think the niche of early-adopters is smaller than the larger market for languages, and the early-adopters are going elsewhere.
The innovator's dilemma, which is really an insight that dates back to Toynbee, and before that Ibn Khaldun, is not so obvious. I am not sure that you have understood it. I suggest reading the book if you are interested, but otherwise I unfortunately don't have so much time at the moment to try to persuade you of what this phenomenon is like and there's limited value to talking about talking rather than having a discussion based on a shared understanding of what this is about.
Nov 29
parent Guillaume Piolat <first.last gmail.com> writes:
On Thursday, 29 November 2018 at 09:18:35 UTC, Laeeth isharc 
wrote:
 The innovator's dilemma, which is really an insight that dates 
 back to Toynbee, and before that Ibn Khaldun, is not so 
 obvious.  I am not sure that you have understood it.  I suggest 
 reading the book if you are interested, but otherwise I 
 unfortunately don't have so much time at the moment to try to 
 persuade you of what this phenomenon is like and there's 
 limited value to talking about talking rather than having a 
 discussion based on a shared understanding of what this is 
 about.
No, indeed I've not read about it (yet, it looks like a tale of our lives). My market has indeed a distribution where the best customer brings 3:1 vs the average, it's not inf:1 like it would be in programming language "customers". "Impulse buy" is predominent too, which does not exist for technical decisions. I understand that a few big players will bring a lot more than hundreds of smaller ones. Especially if the smaller ones keep writing on the D newsgroup :) In my market too, I prefer if D is kind of unpopular! I don't tell details to competitors other than "works for me". And they aren't much interested, which is satisfying. This secrecy has to be balanced by the fact we have to hire, and D being a critical piece of infrastructure I'd like it to simply receive more money, being a small player I contribute what I can to the Foundation - and direct others that came to Dplug to do so.
Nov 29
prev sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Wednesday, 28 November 2018 at 12:48:46 UTC, Laeeth Isharc 
wrote:
 I think that there are different strategies - decent appeal to 
 a broad market and having a very high appeal to a small market 
 (but there has better be something good about your potential 
 customer base ie 'D, if you find VBA too difficult' is probably 
 not a good strategy!).  And you probably don't get to pick 
 which situation you are in, and then one had better realise it 
 and play the game you're in.  The particular kind of market 
 will shape what works - in my business you approach a retail 
 client base differently from regular institutional investors 
 and then the worlds' largest pools of money involved something 
 else again.

 D isn't really marketed and it's definitely not sold.  That's 
 an implicit strategy in itself.
But one doesn't decide to have no strategy (at least if they any common sense!), one simply has no strategy. Unfortunately I think D falls into the latter, certainly not more than "Build it and they will come", irrespective of it effectiveness.
 Actually no less than 3 programmer friends came to (I'm the 
 weirdo-using-D and people are _always_ in disbelief and invent 
 all sorts of reasons not to try) saying they saw an article on 
 D on HN, with "D compilation is slow", and on further 
 examination they didn't read or at best the first paragraph. 
 But they did remember the title. They may rationally think 
 their opinion of D hasn't changed: aren't we highly capable 
 people?
I hope so!
 It doesn't matter what most people think.  It matters what 
 people who are on the fence or using D already a bit think.  Or 
 people who have a lot of problems to which D is in part a 
 solution only they didn't know about or think of D yet.
Then we should try to subtly (for some value of subtlety) make ourselves noticed.
Nov 28
prev sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On Wednesday, 21 November 2018 at 20:51:17 UTC, Walter Bright 
wrote:
 Unfortunately, you're right. The title will leave the 
 impression "D is slow at compiling". You have to carefully read 
 the article to see otherwise, and few will do that.
Well comparative to itself sometimes it is. When you initially write D code you get used to the blazing fast speeds, but when eventually the compilation speed slows down as a project grows then this has a real effect on productivity. Maybe a better title would have been "D compilation sometimes slows down too much", but it wouldn't get as many hits. On the upside, people who read the article - or even just read the comments section, will quickly realize that D's compilation speed is still miles faster than the competition. They might actually try the language. :)
Nov 22
parent reply Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Thursday, 22 November 2018 at 10:51:45 UTC, Andrej Mitrovic 
wrote:

<mega snip>

BTW, it's nice to see again the Secret Squirrel on the forum, in 
these days: welcome back Andrej!

/Paolo
Nov 22
parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On Thursday, 22 November 2018 at 11:16:26 UTC, Paolo Invernizzi 
wrote:
 BTW, it's nice to see again the Secret Squirrel on the forum, 
 in these days: welcome back Andrej!

 /Paolo
Oh hey there too! I'm sorry if I can't recall you, though. ¯\_(ツ)_/¯ I mostly lurk around here these days. But I still use D heavily, at work.
Nov 22
parent Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Thursday, 22 November 2018 at 13:19:58 UTC, Andrej Mitrovic 
wrote:
 On Thursday, 22 November 2018 at 11:16:26 UTC, Paolo Invernizzi 
 wrote:
 BTW, it's nice to see again the Secret Squirrel on the forum, 
 in these days: welcome back Andrej!

 /Paolo
Oh hey there too! I'm sorry if I can't recall you, though. ¯\_(ツ)_/¯
Oh, no problem... eheh
 I mostly lurk around here these days.
Yep, the same...
 But I still use D heavily, at work.
Well, the same here; not so heavily right now, my CTO is not sure anymore about the "case for D", but well, we have just delivered a D (medical) codebase to one of our customer... Let's see... /Paolo
Nov 22
prev sibling parent Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 21 November 2018 at 11:18:22 UTC, Nicholas Wilson 
wrote:
 This is #2 on HN at the moment.
Also on reddit: https://www.reddit.com/r/programming/comments/9z36xg/d_compilation_is_too_slow_and_i_am_forking_the/
Nov 21
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 21 November 2018 at 08:07:52 UTC, Vladimir 
Panteleev wrote:
 https://blog.thecybershadow.net/2018/11/18/d-compilation-is-too-slow-and-i-am-forking-the-compiler/
Very interesting. I'm also currently working on a project to save my bloodstream from the cortisol drip that happens when anything a computer does takes over a second, which these days means waiting for dmd to compile my code so I can see the result of the tests. I'll share more details when I have time. But: one of the things I want to do is a version of this / precompiled headers. I've complained before that compiling std.path with -unittest takes forever (0.5s or so, and most of it due to std.uni). That's a price I pay every time I make a one line change to any file, and the linker hasn't even been invoked yet. Here's the thing: Phobos only changes from one release to the next. Why am I waiting to recompile a read-only file that won't change unless I update the compiler, over and over again? I'd love it if I could precompile Phobos and just use the digested information every time I'm iterating.
Nov 21
parent reply Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Wednesday, 21 November 2018 at 11:35:02 UTC, Atila Neves wrote:
 I'm also currently working on a project to save my bloodstream 
 from the cortisol drip that happens when anything a computer 
 does takes over a second, which these days means waiting for 
 dmd to compile my code so I can see the result of the tests. 
 I'll share more details when I have time.
Looking forward to it!
 But: one of the things I want to do is a version of this / 
 precompiled headers. I've complained before that compiling 
 std.path with -unittest takes forever (0.5s or so, and most of 
 it due to std.uni). That's a price I pay every time I make a 
 one line change to any file, and the linker hasn't even been 
 invoked yet. Here's the thing: Phobos only changes from one 
 release to the next. Why am I waiting to recompile a read-only 
 file that won't change unless I update the compiler, over and 
 over again?
That particular problem is in large part due to that the -unittest switch is not namespaced. I ran into the same issue with -allinst - with std.path, it breaks only if -unittest is also specified. I don't want to compile std.path unit tests, just my own! Have we tried disabling -unittest for modules that aren't on the compiler's command line yet (or, in case of -i, not excluded)?
 I'd love it if I could precompile Phobos and just use the 
 digested information every time I'm iterating.
I agree, it would be nice if we could ship some "precompiled module" files along with Phobos .lib / .so files, but it looks like implementing this feature correctly might be non-trivial. Maybe this hack could be developed further into a more generic "compiler server" idea.
Nov 21
next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Wednesday, 21 November 2018 at 11:58:25 UTC, Vladimir 
Panteleev wrote:
 Have we tried disabling -unittest for modules that aren't on 
 the compiler's command line yet (or, in case of -i, not 
 excluded)?
Not that I know of, thats a great idea!
 Maybe this hack could be developed further into a more generic 
 "compiler server" idea.
Wasn't that Robert's Masters thesis (Dconf 2013(?) presentation)? ;)
Nov 21
next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 21 November 2018 at 13:05:27 UTC, Nicholas Wilson 
wrote:
 On Wednesday, 21 November 2018 at 11:58:25 UTC, Vladimir 
 Panteleev wrote:
 Have we tried disabling -unittest for modules that aren't on 
 the compiler's command line yet (or, in case of -i, not 
 excluded)?
Not that I know of, thats a great idea!
 Maybe this hack could be developed further into a more generic 
 "compiler server" idea.
Wasn't that Robert's Masters thesis (Dconf 2013(?) presentation)? ;)
The main problem with this, is the amount of context a compilers needs. And the current design of DMD does not lend itself splitting out the context. If you wanted you could consider the semantic pass of the compiler as a database, which answers queries such as: - which size does this type have. - which arguments does this function have - can the type A be casted to type B - which conversion function should be invoked for (B)A ? - is this function known to be pure? The data-base containing this information needs to be maintained on the compile-nodes, and that possibly leads to many data-dependencies. Which may degrade the performance of the "compiler server" to the point where it is quicker to do it locally. I am currently working (albeit very slowly due to lack of time and focus) to enable programmers to circumvent slow parts in compiler. When completed this should make a compiler-server unnecessary for some time to come.
Nov 21
prev sibling parent Seb <seb wilzba.ch> writes:
On Wednesday, 21 November 2018 at 13:05:27 UTC, Nicholas Wilson 
wrote:
 On Wednesday, 21 November 2018 at 11:58:25 UTC, Vladimir 
 Panteleev wrote:
 Have we tried disabling -unittest for modules that aren't on 
 the compiler's command line yet (or, in case of -i, not 
 excluded)?
Not that I know of, thats a great idea!
Yes it's sadly a well-known problem e.g. https://github.com/dlang/dmd/pull/8124
Nov 21
prev sibling parent Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 21 November 2018 at 11:58:25 UTC, Vladimir 
Panteleev wrote:
 On Wednesday, 21 November 2018 at 11:35:02 UTC, Atila Neves 
 wrote:
 [...]
Looking forward to it!
 [...]
That particular problem is in large part due to that the -unittest switch is not namespaced. I ran into the same issue with -allinst - with std.path, it breaks only if -unittest is also specified. I don't want to compile std.path unit tests, just my own! Have we tried disabling -unittest for modules that aren't on the compiler's command line yet (or, in case of -i, not excluded)?
I'm not sure what the real issue is here or what the solution would be. There was a PR to fix it that was closed without merging.
Nov 21