www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - SAOC Experience Report: Porting a fork-based GC

reply Mike Parker <aldacron gmail.com> writes:
Francesco Mecca ha written an experience report for the D Blog 
about his SAOC 2018 project, porting Leandro Lucarella's old GC 
from D1 to D2.

The blog:
https://dlang.org/blog/wp-admin/post.php?post=2148&action=edit

Reddit:
https://www.reddit.com/r/programming/comments/cgdk1r/symmetry_autumn_of_code_experience_report_porting/
Jul 22
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
https://dlang.org/blog/2019/07/22/symmetry-autumn-of-code-experience-report-porting-a-fork-based-gc/
Jul 22
parent Mike Parker <aldacron gmail.com> writes:
On Monday, 22 July 2019 at 14:04:32 UTC, rikki cattermole wrote:
 https://dlang.org/blog/2019/07/22/symmetry-autumn-of-code-experience-report-porting-a-fork-based-gc/
Thanks!
Jul 22
prev sibling next sibling parent reply a11e99z <black80 bk.ru> writes:
On Monday, 22 July 2019 at 14:03:15 UTC, Mike Parker wrote:
 Francesco Mecca ha written an experience report for the D Blog 
 about his SAOC 2018 project, porting Leandro Lucarella's old GC 
 from D1 to D2.

 The blog:
 https://dlang.org/blog/wp-admin/post.php?post=2148&action=edit

 Reddit:
 https://www.reddit.com/r/programming/comments/cgdk1r/symmetry_autumn_of_code_experience_report_porting/
interesting idea do Mark phase through fork process. I want more technical details: - about Windows: will such idea work on Windows? does Windows support similar technique? (VirtualQueryEx/ReadProcessMemory/mmap/or_something) - can GC-process run full time and just exchange with parent process about changes in memory configuration? or GC-process can not to see changes in parent process after fork? I mean GC-process can see snapshots of parent-process memory in realtime when it has just readonly-access. - does GC-process uses some AA for building graph of objects or set Mark-bits in situ? in last case COW makes copies so memory differs and GC-process should be forked every GC-cycle. - digits in PR benchmark scary me: max pause 969ms or 260ms. why not to use Golang-GC? Go-GC is non-compacting/non-moving GC that has STW(stop the world) ~10ms and Google can spend money to build best GC that can help to D-users too.
Jul 22
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 23/07/2019 3:58 AM, a11e99z wrote:
 On Monday, 22 July 2019 at 14:03:15 UTC, Mike Parker wrote:
 Francesco Mecca ha written an experience report for the D Blog about 
 his SAOC 2018 project, porting Leandro Lucarella's old GC from D1 to D2.

 The blog:
 https://dlang.org/blog/wp-admin/post.php?post=2148&action=edit

 Reddit:
 https://www.reddit.com/r/programming/comments/cgdk1r/symmetry_autumn_of_code_experi
nce_report_porting/ 
interesting idea do Mark phase through fork process. I want more technical details: - about Windows: will such idea work on Windows? does Windows support similar technique? (VirtualQueryEx/ReadProcessMemory/mmap/or_something)
The problem with Windows is not so much reading another process memory, is the fact that you cannot create another process which is copied from your own memory space AND be able to have live threads running in it. What we can do (which will be good for things like audio and game engines) is make the GC only suspend the current thread and read from a snapshot of the heap space of the process. This we can do. Unfortunately I remembered that marking is a bit more complex than just locating memory a bit too late tonight. No wonder I never tried before now to implement this (I have evaluated it). I may have another try at it, if somebody can give me pointers on where the actual reading occurs in the GC. Because it has left me a bit stumped.
Jul 22
parent reply Francesco Mecca <me francescomecca.eu> writes:
On Monday, 22 July 2019 at 16:52:08 UTC, rikki cattermole wrote:
 I may have another try at it, if somebody can give me pointers 
 on where the actual reading occurs in the GC. Because it has 
 left me a bit stumped.
Come talk in the IRC channel. I am there and people are super helpful.
Jul 23
parent rikki cattermole <rikki cattermole.co.nz> writes:
On 23/07/2019 9:50 PM, Francesco Mecca wrote:
 On Monday, 22 July 2019 at 16:52:08 UTC, rikki cattermole wrote:
 I may have another try at it, if somebody can give me pointers on 
 where the actual reading occurs in the GC. Because it has left me a 
 bit stumped.
Come talk in the IRC channel. I am there and people are super helpful.
I'm one of those helpful people :)
Jul 23
prev sibling parent reply Francesco Mecca <me francescomecca.eu> writes:
On Monday, 22 July 2019 at 15:58:08 UTC, a11e99z wrote:
 On Monday, 22 July 2019 at 14:03:15 UTC, Mike Parker wrote:
 Francesco Mecca ha written an experience report for the D Blog 
 about his SAOC 2018 project, porting Leandro Lucarella's old 
 GC from D1 to D2.

 The blog:
 https://dlang.org/blog/wp-admin/post.php?post=2148&action=edit

 Reddit:
 https://www.reddit.com/r/programming/comments/cgdk1r/symmetry_autumn_of_code_experience_report_porting/
interesting idea do Mark phase through fork process. I want more technical details: - about Windows: will such idea work on Windows? does Windows support similar technique? (VirtualQueryEx/ReadProcessMemory/mmap/or_something)
I have no idea about Windows. You can read about it here: https://rainers.github.io/visuald/druntime/concurrentgc.html
 - can GC-process run full time and just exchange with parent 
 process about changes in memory configuration? or GC-process 
 can not to see changes in parent process after fork? I mean 
 GC-process can see snapshots of parent-process memory in 
 realtime when it has just readonly-access.
It could given that the bitarrays that are needed for the algorithm could be mmapped. In order to have a children running full time we would need a way to signal that a new allocation is taking place and to start a mark phase. We could experiment in this direction (it shouldn't be much work at the current state) but to me it seems overkill given that it is related to the mark routines only and not the whole garbage collection cycle.
 - does GC-process uses some AA for building graph of objects or 
 set Mark-bits in situ? in last case COW makes copies so memory 
 differs and GC-process should be forked every GC-cycle.
Last case, but the needed bits could be mmapped in order to be shared. Impact is minimal.
 - digits in PR benchmark scary me: max pause 969ms or 260ms. 
 why not to use Golang-GC? Go-GC is non-compacting/non-moving GC 
 that has STW(stop the world) ~10ms and Google can spend money 
 to build best GC that can help to D-users too.
First of all I am not aware of the go-gc being provided as a library, it should be tightly coupled to the go runtime. Also, I think it uses write barriers and D doesn't have them. Moreover, the go-gc runs in background all the time, trashing the cache and stealing cpu cycles. I don't know if the situation has changed but that is a major compromise. Regarding dlang I would suggest to stop worrying about the GC and profile. The GC is highly deterministic (a property that few GC out there share) so that the user knows when it runs. Currently there are many facilities for profiling (and more could be added) so that bottlenecks caused by the GC can be solved on a case by case basis. Post such as this: https://blog.cloudflare.com/go-dont-collect-my-garbage/ shows that in real world scenarios people try to tweak the many knobs that GC implementers provide in order to adjust overall performance. The main difference is that in D you can isolate the few cases that are impacted by the GC and avoid garbage collection in that path.
Jul 23
parent a11e99z <black80 bk.ru> writes:
On Tuesday, 23 July 2019 at 11:28:35 UTC, Francesco Mecca wrote:
 On Monday, 22 July 2019 at 15:58:08 UTC, a11e99z wrote:
 On Monday, 22 July 2019 at 14:03:15 UTC, Mike Parker wrote:
 Francesco Mecca ha written an experience report for the D
thanks for answers
Jul 23
prev sibling next sibling parent JN <666total wp.pl> writes:
On Monday, 22 July 2019 at 14:03:15 UTC, Mike Parker wrote:
 Francesco Mecca ha written an experience report for the D Blog 
 about his SAOC 2018 project, porting Leandro Lucarella's old GC 
 from D1 to D2.

 The blog:
 https://dlang.org/blog/wp-admin/post.php?post=2148&action=edit

 Reddit:
 https://www.reddit.com/r/programming/comments/cgdk1r/symmetry_autumn_of_code_experience_report_porting/
Should have included mention of D in the title
Jul 22
prev sibling next sibling parent a11e99z <black80 bk.ru> writes:
On Monday, 22 July 2019 at 14:03:15 UTC, Mike Parker wrote:
 Francesco Mecca ha written an experience report for the D Blog 
 about his SAOC 2018 project, porting Leandro Lucarella's old GC 
 from D1 to D2.

 The blog:
 https://dlang.org/blog/wp-admin/post.php?post=2148&action=edit
OT: this link try to login me in WordPress and to redirect to D-blog probably cuz "wp-admin" and "&action=edit" in URL
Jul 22
prev sibling parent reply Meta <jared771 gmail.com> writes:
On Monday, 22 July 2019 at 14:03:15 UTC, Mike Parker wrote:
 Francesco Mecca ha written an experience report for the D Blog 
 about his SAOC 2018 project, porting Leandro Lucarella's old GC 
 from D1 to D2.

 The blog:
 https://dlang.org/blog/wp-admin/post.php?post=2148&action=edit

 Reddit:
 https://www.reddit.com/r/programming/comments/cgdk1r/symmetry_autumn_of_code_experience_report_porting/
A pull request to the D runtime was my final milestone. I was ready at the beginning of February, but I started to procrastinate. I’d had no previous communication with any of the reviewers and I was timorous about engaging with them. I spent a lot of time refactoring my code back and forth and delaying my pull request. At a certain point, I even considered abandoning the final milestone and providing the GC as a library. In the meantime, Rainer Scheutze published a threaded implementation of the mark phase that reduced the mark time in the GC and I lost faith in my project. This seems like a major failure in the process that this was allowed to happen - good work almost went abandoned. How can we prevent this in future SAoC/GSoC? Without knowing what the mentor did/didn't do, an obvious answer seems like there should be a follow-up to ensure that the work done is actually getting in to the compiler/runtime/etc. To go so far and trip right at the finish line is unfortunate (glad to see that a PR is now open).
Jul 22
next sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Monday, 22 July 2019 at 20:57:19 UTC, Meta wrote:
 [snip]

 This seems like a major failure in the process that this was 
 allowed to happen - good work almost went abandoned. How can we 
 prevent this in future SAoC/GSoC? Without knowing what the 
 mentor did/didn't do, an obvious answer seems like there should 
 be a follow-up to ensure that the work done is actually getting 
 in to the compiler/runtime/etc. To go so far and trip right at 
 the finish line is unfortunate (glad to see that a PR is now 
 open).
I've been really liking the progress updates that the GSoC students are doing this year.
Jul 22
prev sibling parent Francesco Mecca <me francescomecca.eu> writes:
On Monday, 22 July 2019 at 20:57:19 UTC, Meta wrote:
 On Monday, 22 July 2019 at 14:03:15 UTC, Mike Parker wrote:
 [...]
A pull request to the D runtime was my final milestone. I was ready at the beginning of February, but I started to procrastinate. I’d had no previous communication with any of the reviewers and I was timorous about engaging with them. I spent a lot of time refactoring my code back and forth and delaying my pull request. At a certain point, I even considered abandoning the final milestone and providing the GC as a library. In the meantime, Rainer Scheutze published a threaded implementation of the mark phase that reduced the mark time in the GC and I lost faith in my project. [...]
at the dconf I had the possibility to talk about this with many people and I think that so far we are witnessing major improvements regarding the GSOC students.
Jul 23