www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - [Article] D's Garbage Collector Problem

reply "Ryan Voots" <simcop2387 simcop2387.info> writes:
An article today about the garbage collector and the ensuing 
discussion from it

http://pointersgonewild.wordpress.com/2014/09/09/ds-garbage-collector-problem/

https://news.ycombinator.com/item?id=8297091

http://www.reddit.com/r/programming/comments/2g03af/ds_garbage_collector_problem/
Sep 10 2014
parent reply "Joakim" <dlang joakim.airpost.net> writes:
On Wednesday, 10 September 2014 at 18:58:58 UTC, Ryan Voots wrote:
 An article today about the garbage collector and the ensuing 
 discussion from it

 http://pointersgonewild.wordpress.com/2014/09/09/ds-garbage-collector-problem/

 https://news.ycombinator.com/item?id=8297091

 http://www.reddit.com/r/programming/comments/2g03af/ds_garbage_collector_problem/
Orvid, where's that new GC when we need it? ;)
Sep 10 2014
parent reply "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Wednesday, 10 September 2014 at 20:01:40 UTC, Joakim wrote:
 Orvid, where's that new GC when we need it? ;)
Andrei posted this in the reddit thread: "OK that does it. I'm going to redesign and implement D's tracing garbage collector using the core allocator I wrote a short while ago. Fatefully Walter Bright and I were talking about the GC over dinner last night (I'm at cppcon in Seattle!) and I figured if this is what matters for D, I'll have to do it. And it does matter. It's actually not that difficult especially given that we have a solid allocator backend."
Sep 10 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/10/14, 1:11 PM, Peter Alexander wrote:
 On Wednesday, 10 September 2014 at 20:01:40 UTC, Joakim wrote:
 Orvid, where's that new GC when we need it? ;)
Andrei posted this in the reddit thread: "OK that does it. I'm going to redesign and implement D's tracing garbage collector using the core allocator I wrote a short while ago. Fatefully Walter Bright and I were talking about the GC over dinner last night (I'm at cppcon in Seattle!) and I figured if this is what matters for D, I'll have to do it. And it does matter. It's actually not that difficult especially given that we have a solid allocator backend."
It was about time! -- Andrei
Sep 10 2014
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 11 September 2014 at 00:27:40 UTC, Andrei
Alexandrescu wrote:
 It was about time! -- Andrei
Tychus, is that you ? http://www.blizzcon2013.com/wp-content/uploads/2012/09/starcraft_ii_beta_time.jpg
Sep 10 2014
parent reply "Kagamin" <spam here.lot> writes:
There are various api the compiler use to allocate from the GC. 
Some do not specify if the allocated memory contains pointers or 
not, and none do specify the type qualifier of the memory.
Is it true about pointers? Which functions? And why type qualifiers matter?
Sep 11 2014
parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Thu, 11 Sep 2014 14:30:05 +0000
schrieb "Kagamin" <spam here.lot>:

There are various api the compiler use to allocate from the GC. 
Some do not specify if the allocated memory contains pointers or 
not, and none do specify the type qualifier of the memory.
Is it true about pointers? Which functions? And why type qualifiers matter?
Immutable data structures cannot have pointers changed or set to null. Also they can only reference other immutable data. This means that they form sort of a big blob that is kept alive by one or more pointers to it, but the GC never needs to check the immutable pointers inside of it. Shared/unshared may affect implementations that provide thread local GC. E.g. only shared data needs to be handled by a global stop the world GC. I'm not sure though. -- Marco
Sep 11 2014
next sibling parent reply "Kagamin" <spam here.lot> writes:
On Thursday, 11 September 2014 at 16:26:37 UTC, Marco Leise wrote:
 Immutable data structures cannot have pointers changed or set
 to null. Also they can only reference other immutable data.
 This means that they form sort of a big blob that is kept
 alive by one or more pointers to it, but the GC never needs
 to check the immutable pointers inside of it.
GC still should free unreachable parts of that blob.
 Shared/unshared may affect implementations that provide thread
 local GC. E.g. only shared data needs to be handled by a
 global stop the world GC. I'm not sure though.
Can get in a way, when you need a thread-local immutable object.
Sep 13 2014
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 13 September 2014 at 18:06:47 UTC, Kagamin wrote:
 On Thursday, 11 September 2014 at 16:26:37 UTC, Marco Leise 
 wrote:
 Immutable data structures cannot have pointers changed or set
 to null. Also they can only reference other immutable data.
 This means that they form sort of a big blob that is kept
 alive by one or more pointers to it, but the GC never needs
 to check the immutable pointers inside of it.
GC still should free unreachable parts of that blob.
 Shared/unshared may affect implementations that provide thread
 local GC. E.g. only shared data needs to be handled by a
 global stop the world GC. I'm not sure though.
Can get in a way, when you need a thread-local immutable object.
Hmm... when is that necessary? Wouldn't `const` be enough? AFAICS, `immutable` is mostly useful for shared data.
Sep 13 2014
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 09/13/2014 09:07 PM, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
<schuetzm gmx.net>" wrote:
 Wouldn't `const` be enough? AFAICS, `immutable` is mostly useful for
 shared data.
const prevents modification, but does not guarantee that no modification takes place elsewhere. I.e. it enables fewer program transformations.
Sep 13 2014
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 13 September 2014 at 19:20:51 UTC, Timon Gehr wrote:
 On 09/13/2014 09:07 PM, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" wrote:
 Wouldn't `const` be enough? AFAICS, `immutable` is mostly 
 useful for
 shared data.
const prevents modification, but does not guarantee that no modification takes place elsewhere. I.e. it enables fewer program transformations.
But as it is thread-local, is it really relevant (for optimization)? I can see two situations where it can potentially matter: pointer parameter aliasing, and calls to function/delegate calls that might modify a value referenced by a pointer. In practice the latter is probably not so important for optimizations, because the compiler would most likely have a hard time caching a value across a function call because of register pressure, so a reload would be necessary anyway. I don't know about parameter aliasing, but I guess it only makes a difference in corner cases. In these cases, it's easy to forbid it by contract (as slice copying does).
Sep 14 2014
prev sibling parent "Kagamin" <spam here.lot> writes:
And strong purity.
Sep 13 2014
prev sibling parent "Kagamin" <spam here.lot> writes:
On Thursday, 11 September 2014 at 16:26:37 UTC, Marco Leise wrote:
 Shared/unshared may affect implementations that provide thread
 local GC. E.g. only shared data needs to be handled by a
 global stop the world GC. I'm not sure though.
Also a certain discipline is required to construct shared/immutable object from the start instead of casting from thread-local mutable. And then how to add an object to a shared collection? //callable without lock void add(shared Object) shared; //allocated thread-local object void add(Object); //but unshared collection contains unshared objects void add(shared Object); //thread-local object again void add(Object) shared; ?
Sep 13 2014
prev sibling next sibling parent reply "Chris" <wendlec tcd.ie> writes:
On Thursday, 11 September 2014 at 00:27:40 UTC, Andrei 
Alexandrescu wrote:
 On 9/10/14, 1:11 PM, Peter Alexander wrote:
 On Wednesday, 10 September 2014 at 20:01:40 UTC, Joakim wrote:
 Orvid, where's that new GC when we need it? ;)
Andrei posted this in the reddit thread: "OK that does it. I'm going to redesign and implement D's tracing garbage collector using the core allocator I wrote a short while ago. Fatefully Walter Bright and I were talking about the GC over dinner last night (I'm at cppcon in Seattle!) and I figured if this is what matters for D, I'll have to do it. And it does matter. It's actually not that difficult especially given that we have a solid allocator backend."
It was about time! -- Andrei
Thank you very much for tackling this problem. GC hasn't been a problem for me (yet!), but the article is very convincing. Writing an JIT compiler whose performance is as good as or even better than C(++)'s is something one should be able to do in D, there's no doubt about it. The author of the aforementioned article apologizes for the "bad publicity" he may have caused. On the contrary, I think he has done D a big favor. Through the attention his article got, the (programming) world knows now that D will get a much better GC, which will (hopefully) bring D back on the radar of those who had already discarded it because of its poor GC. Also, unlike the corporate world, we should not be afraid of "bad publicity". If there is a real problem, it's better to talk about it and fix it than to cover it up and drag it along.
Sep 11 2014
parent reply ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Thu, 11 Sep 2014 09:29:41 +0000
Chris via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 he has
she has. ;-)
Sep 11 2014
parent reply "Chris" <wendlec tcd.ie> writes:
On Thursday, 11 September 2014 at 09:37:04 UTC, ketmar via 
Digitalmars-d wrote:
 On Thu, 11 Sep 2014 09:29:41 +0000
 Chris via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 he has
she has. ;-)
Japers, I wasn't sure, if it was he or she and just gambled on a 50% chance :) My apologies. PS and OT I hope Higgs will become a long term project. Imagine, you could combine Higgs and vibe.d. You could write your own D extensions for server side JS. In this way you could have projects where anyone who knows JS could contribute (good bye PHP), no need to know D. I have a test version of a LuaD + Lua Server Pages + vibe.d server. Works very well. JS is even more wide spread than Lua (albeit inferior as a language).
Sep 11 2014
parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 11/09/2014 10:05 p.m., Chris wrote:
 On Thursday, 11 September 2014 at 09:37:04 UTC, ketmar via Digitalmars-d
 wrote:
 On Thu, 11 Sep 2014 09:29:41 +0000
 Chris via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 he has
she has. ;-)
Japers, I wasn't sure, if it was he or she and just gambled on a 50% chance :) My apologies. PS and OT I hope Higgs will become a long term project. Imagine, you could combine Higgs and vibe.d. You could write your own D extensions for server side JS. In this way you could have projects where anyone who knows JS could contribute (good bye PHP), no need to know D. I have a test version of a LuaD + Lua Server Pages + vibe.d server. Works very well. JS is even more wide spread than Lua (albeit inferior as a language).
*starts drooling at the idea of using Jade for Cmsed* Would seriously fix so many problems with templates. Any chance I could get a copy of your test for LuaD + Lua Server Pages? I may want to investigate that angle as well.
Sep 11 2014
parent reply "Chris" <wendlec tcd.ie> writes:
On Thursday, 11 September 2014 at 10:23:56 UTC, Rikki Cattermole 
wrote:
 On 11/09/2014 10:05 p.m., Chris wrote:
 On Thursday, 11 September 2014 at 09:37:04 UTC, ketmar via 
 Digitalmars-d
 wrote:
 On Thu, 11 Sep 2014 09:29:41 +0000
 Chris via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 he has
she has. ;-)
Japers, I wasn't sure, if it was he or she and just gambled on a 50% chance :) My apologies. PS and OT I hope Higgs will become a long term project. Imagine, you could combine Higgs and vibe.d. You could write your own D extensions for server side JS. In this way you could have projects where anyone who knows JS could contribute (good bye PHP), no need to know D. I have a test version of a LuaD + Lua Server Pages + vibe.d server. Works very well. JS is even more wide spread than Lua (albeit inferior as a language).
*starts drooling at the idea of using Jade for Cmsed* Would seriously fix so many problems with templates. Any chance I could get a copy of your test for LuaD + Lua Server Pages? I may want to investigate that angle as well.
Of course. Once I have a "clean" version. The first test version is full of embarrassing hit-and-miss code, commented blocks etc. But you could probably glue one together yourself. 1. get Open Luasp http://www.luasp.org/ 2. get LuaD 3. create a vibe.d project (a normal default project will do, for testing at least) 4. link your project to libluad and liblualsp (or lualsp.so) In you vibe.d code route to your lua handling function shared static this() { [...] router.any("*", &luaTest); [...] } void luaTest(HTTPServerRequest req, HTTPServerResponse res) { // do your lualsp / LuaD thing } Then you have basically a few options. Use Lua code as in test.lua: repuire("lualsp") dofile_lsp("example.html") in (Lua)D: lua.doFile("test.lua") or lua.doString("require('lualsp')"); lua.doString("dofile_lsp('public/example1.html')"); or (what I do), call the functions provided by lualsp directly extern (C) int luaopen_lualsp(lua_State *L); extern (C) int luaL_do_lsp_file(lua_State* L,const char* filename); [etc....] Or a combination of the above. Just have a look at the various methods provided by both LuaD and lualsp. Note: lualsp writes to stdout, make sure you use std.c.stdio.stdout to grab the output. Not the one provided in Phobos' std.stdio Excerpt from dub.json "libs": [ "luad", "lua5.1", "lualsp" ], Make sure you use Lua 5.1. LuaD (afsaik) only supports 5.1 as of now. It works like PHP (echo, print etc). I put this together in order to avoid PHP in the future. Lua is a sound language and you can expect developers to be familiar with it.
Sep 11 2014
parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 11/09/2014 11:13 p.m., Chris wrote:
 On Thursday, 11 September 2014 at 10:23:56 UTC, Rikki Cattermole wrote:
 On 11/09/2014 10:05 p.m., Chris wrote:
 On Thursday, 11 September 2014 at 09:37:04 UTC, ketmar via Digitalmars-d
 wrote:
 On Thu, 11 Sep 2014 09:29:41 +0000
 Chris via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 he has
she has. ;-)
Japers, I wasn't sure, if it was he or she and just gambled on a 50% chance :) My apologies. PS and OT I hope Higgs will become a long term project. Imagine, you could combine Higgs and vibe.d. You could write your own D extensions for server side JS. In this way you could have projects where anyone who knows JS could contribute (good bye PHP), no need to know D. I have a test version of a LuaD + Lua Server Pages + vibe.d server. Works very well. JS is even more wide spread than Lua (albeit inferior as a language).
*starts drooling at the idea of using Jade for Cmsed* Would seriously fix so many problems with templates. Any chance I could get a copy of your test for LuaD + Lua Server Pages? I may want to investigate that angle as well.
Of course. Once I have a "clean" version. The first test version is full of embarrassing hit-and-miss code, commented blocks etc.
Personally I'm not worried. I wouldn't be able to use it in any form.
 But you could probably glue one together yourself.

 1. get Open Luasp http://www.luasp.org/
 2. get LuaD
 3. create a vibe.d project (a normal default project will do, for
 testing at least)
 4. link your project to libluad and liblualsp (or lualsp.so)

 In you vibe.d code route to your lua handling function

 shared static this() {
    [...]
    router.any("*", &luaTest);
    [...]
 }

 void luaTest(HTTPServerRequest req, HTTPServerResponse res) {
    // do your lualsp / LuaD thing
 }

 Then you have basically a few options. Use Lua code as in

 test.lua:

 repuire("lualsp")
 dofile_lsp("example.html")

 in (Lua)D:

 lua.doFile("test.lua")

 or

 lua.doString("require('lualsp')");
 lua.doString("dofile_lsp('public/example1.html')");

 or (what I do), call the functions provided by lualsp directly

 extern (C) int luaopen_lualsp(lua_State *L);
 extern (C) int luaL_do_lsp_file(lua_State* L,const char* filename);
 [etc....]

 Or a combination of the above.

 Just have a look at the various methods provided by both LuaD and lualsp.

 Note: lualsp writes to stdout, make sure you use

 std.c.stdio.stdout

 to grab the output. Not the one provided in Phobos' std.stdio

 Excerpt from dub.json

 "libs": [
      "luad",
      "lua5.1",
      "lualsp"
      ],

 Make sure you use Lua 5.1. LuaD (afsaik) only supports 5.1 as of now.

 It works like PHP (echo, print etc). I put this together in order to
 avoid PHP in the future. Lua is a sound language and you can expect
 developers to be familiar with it.
I see hmm, I may compile a custom lua dll then with lanes (think threading). Well maybe, we shall see. Might also be a good idea to hook it up to my skeleton tool[0] to enable using e.g. gists for require's. [0] https://github.com/rikkimax/skeleton
Sep 11 2014
next sibling parent "Chris" <wendlec tcd.ie> writes:
On Thursday, 11 September 2014 at 11:28:38 UTC, Rikki Cattermole 
wrote:
 I see hmm, I may compile a custom lua dll then with lanes 
 (think threading). Well maybe, we shall see.
 Might also be a good idea to hook it up to my skeleton tool[0] 
 to enable using e.g. gists for require's.

 [0] https://github.com/rikkimax/skeleton
Maybe I can set up a simple test version for you, when I find the time. Can't promise but I'll try. One thing I haven't looked into yet is a clever way of handling the output in stdout. At the moment, I write to a temporary file ("test.txt") I read in again.
Sep 11 2014
prev sibling parent reply "Chris" <wendlec tcd.ie> writes:
On Thursday, 11 September 2014 at 11:28:38 UTC, Rikki Cattermole 
wrote:

Rikki, I have a basic LuaD / vibe.d / Lua LSP server now. I 
needed actually less code than expected. If you give me your 
email address, I can send it to you. It has a simple form example.
Sep 12 2014
parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 12/09/2014 11:30 p.m., Chris wrote:
 On Thursday, 11 September 2014 at 11:28:38 UTC, Rikki Cattermole wrote:

 Rikki, I have a basic LuaD / vibe.d / Lua LSP server now. I needed
 actually less code than expected. If you give me your email address, I
 can send it to you. It has a simple form example.
Okay sweet. alphaglosined /at/ gmail /dot/ com
Sep 12 2014
parent "Chris" <wendlec tcd.ie> writes:
On Friday, 12 September 2014 at 11:46:08 UTC, Rikki Cattermole 
wrote:
 On 12/09/2014 11:30 p.m., Chris wrote:
 On Thursday, 11 September 2014 at 11:28:38 UTC, Rikki 
 Cattermole wrote:

 Rikki, I have a basic LuaD / vibe.d / Lua LSP server now. I 
 needed
 actually less code than expected. If you give me your email 
 address, I
 can send it to you. It has a simple form example.
Okay sweet. alphaglosined /at/ gmail /dot/ com
Ok, I've sent it.
Sep 12 2014
prev sibling parent "Bastiaan Veelo" <Bastiaan Veelo.net> writes:
Yay!
Sep 11 2014