www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - GC API: What can change for precise scanning?

reply "dsimcha" <dsimcha yahoo.com> writes:
Now that the compiler infrastructure has been implemented, I've 
gotten busy figuring out how to make D's default GC precise.  As 
a first attempt, I think I'm going to adapt my original solution 
from http://d.puremagic.com/issues/show_bug.cgi?id=3463 since 
it's simple and it works except that there previously was no 
clean way to get the offset info into the GC.  As Walter pointed 
out in another thread, the GCInfo template is allowed to 
instantiate to data instead of a function.  IMHO unless/until 
major architectural changes to the GC are made that require a 
function pointer, there's no point in adding this indirection.

I started working on this and I ran into a roadblock.  I need to 
know what parts of the GC API are allowed to change, and discuss 
how to abstract away the implementation of it from the GC API.  I 
assume the stuff in core.memory needs to stay mostly the same, 
though I guess we would need to add a setType() function that 
takes a pointer into a block of memory and a TypeInfo object and 
changes how the GC interprets the bits in the block.

In gc.d, we define a bunch of extern(C) functions and the proxy 
thing.  Since we've given up on the idea of swapping precise GCs 
at link time, can I just rip out all this unnecesary indirection? 
  If not, is it ok to change some of these signatures?  I 
definitely want to avoid allocating (requiring the GC lock) and 
then calling a function to set the type (requiring another lock 
acquisition) so the signature of malloc(), etc. needs to change 
somewhere.

More generally, what is the intended way to get GCInfo pointers 
from TypeInfo into the guts of the GC where they can be acted on?
Apr 17 2012
next sibling parent reply deadalnix <deadalnix gmail.com> writes:
Le 18/04/2012 02:36, dsimcha a écrit :
 Now that the compiler infrastructure has been implemented, I've gotten
 busy figuring out how to make D's default GC precise. As a first
 attempt, I think I'm going to adapt my original solution from
 http://d.puremagic.com/issues/show_bug.cgi?id=3463 since it's simple and
 it works except that there previously was no clean way to get the offset
 info into the GC. As Walter pointed out in another thread, the GCInfo
 template is allowed to instantiate to data instead of a function. IMHO
 unless/until major architectural changes to the GC are made that require
 a function pointer, there's no point in adding this indirection.

 I started working on this and I ran into a roadblock. I need to know
 what parts of the GC API are allowed to change, and discuss how to
 abstract away the implementation of it from the GC API. I assume the
 stuff in core.memory needs to stay mostly the same, though I guess we
 would need to add a setType() function that takes a pointer into a block
 of memory and a TypeInfo object and changes how the GC interprets the
 bits in the block.

 In gc.d, we define a bunch of extern(C) functions and the proxy thing.
 Since we've given up on the idea of swapping precise GCs at link time,
 can I just rip out all this unnecesary indirection? If not, is it ok to
 change some of these signatures? I definitely want to avoid allocating
 (requiring the GC lock) and then calling a function to set the type
 (requiring another lock acquisition) so the signature of malloc(), etc.
 needs to change somewhere.

 More generally, what is the intended way to get GCInfo pointers from
 TypeInfo into the guts of the GC where they can be acted on?

I guess that the flag to indicate if some piece of memory may have pointer can go away. I think you certainly can remove all indirection. Additionally, I wonder why most of theses functions are extern(C).
Apr 18 2012
next sibling parent =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <xtzgzorex gmail.com> writes:
On 18-04-2012 11:56, deadalnix wrote:
 Le 18/04/2012 02:36, dsimcha a écrit :
 Now that the compiler infrastructure has been implemented, I've gotten
 busy figuring out how to make D's default GC precise. As a first
 attempt, I think I'm going to adapt my original solution from
 http://d.puremagic.com/issues/show_bug.cgi?id=3463 since it's simple and
 it works except that there previously was no clean way to get the offset
 info into the GC. As Walter pointed out in another thread, the GCInfo
 template is allowed to instantiate to data instead of a function. IMHO
 unless/until major architectural changes to the GC are made that require
 a function pointer, there's no point in adding this indirection.

 I started working on this and I ran into a roadblock. I need to know
 what parts of the GC API are allowed to change, and discuss how to
 abstract away the implementation of it from the GC API. I assume the
 stuff in core.memory needs to stay mostly the same, though I guess we
 would need to add a setType() function that takes a pointer into a block
 of memory and a TypeInfo object and changes how the GC interprets the
 bits in the block.

 In gc.d, we define a bunch of extern(C) functions and the proxy thing.
 Since we've given up on the idea of swapping precise GCs at link time,
 can I just rip out all this unnecesary indirection? If not, is it ok to
 change some of these signatures? I definitely want to avoid allocating
 (requiring the GC lock) and then calling a function to set the type
 (requiring another lock acquisition) so the signature of malloc(), etc.
 needs to change somewhere.

 More generally, what is the intended way to get GCInfo pointers from
 TypeInfo into the guts of the GC where they can be acted on?

I guess that the flag to indicate if some piece of memory may have pointer can go away.

+1. This is useless if we're going to use bitmaps or similar.
 I think you certainly can remove all indirection. Additionally, I wonder
 why most of theses functions are extern(C).

-- - Alex
Apr 18 2012
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/18/2012 2:56 AM, deadalnix wrote:
 I think you certainly can remove all indirection. Additionally, I wonder why
 most of theses functions are extern(C).

The purpose of the indirection is so that DLLs in Windows can share a gc instance, rather than having two instances fight each other.
Apr 18 2012
prev sibling parent reply deadalnix <deadalnix gmail.com> writes:
Le 18/04/2012 20:53, Sean Kelly a écrit :
 On Apr 18, 2012, at 2:56 AM, deadalnix wrote:
 I think you certainly can remove all indirection. Additionally, I wonder why
most of theses functions are extern(C).

So the GC implementation is opaque and the GC can therefore be chosen at link-time. Similar to how the compiler runtime code hides behind a raft of extern© functions.

I know, but this is now impossible anyway because of the modification of TypeInfo anyway.
Apr 18 2012
parent reply deadalnix <deadalnix gmail.com> writes:
Le 19/04/2012 02:08, Sean Kelly a écrit :
 On Apr 18, 2012, at 4:02 PM, deadalnix wrote:

 Le 18/04/2012 20:53, Sean Kelly a écrit :
 On Apr 18, 2012, at 2:56 AM, deadalnix wrote:
 I think you certainly can remove all indirection. Additionally, I wonder why
most of theses functions are extern(C).

So the GC implementation is opaque and the GC can therefore be chosen at link-time. Similar to how the compiler runtime code hides behind a raft of extern© functions.

I know, but this is now impossible anyway because of the modification of TypeInfo anyway.

I'm not sure I follow. Are you saying the change breaks having the GC behind extern C functions? How?

No it doesn't break GC behind C functions. It break the possibility of changing GC at link time, because different GC needs different data generated in TypeInfo. So the indirection become useless.
Apr 18 2012
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 04/19/2012 02:21 AM, deadalnix wrote:
 Le 19/04/2012 02:08, Sean Kelly a écrit :
 On Apr 18, 2012, at 4:02 PM, deadalnix wrote:

 Le 18/04/2012 20:53, Sean Kelly a écrit :
 On Apr 18, 2012, at 2:56 AM, deadalnix wrote:
 I think you certainly can remove all indirection. Additionally, I
 wonder why most of theses functions are extern(C).

So the GC implementation is opaque and the GC can therefore be chosen at link-time. Similar to how the compiler runtime code hides behind a raft of extern© functions.

I know, but this is now impossible anyway because of the modification of TypeInfo anyway.

I'm not sure I follow. Are you saying the change breaks having the GC behind extern C functions? How?

No it doesn't break GC behind C functions. It break the possibility of changing GC at link time, because different GC needs different data generated in TypeInfo. So the indirection become useless.

The indirection is there for better shared library support.
Apr 18 2012
prev sibling next sibling parent =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <xtzgzorex gmail.com> writes:
On 18-04-2012 02:36, dsimcha wrote:
 Now that the compiler infrastructure has been implemented, I've gotten
 busy figuring out how to make D's default GC precise. As a first
 attempt, I think I'm going to adapt my original solution from
 http://d.puremagic.com/issues/show_bug.cgi?id=3463 since it's simple and
 it works except that there previously was no clean way to get the offset
 info into the GC. As Walter pointed out in another thread, the GCInfo
 template is allowed to instantiate to data instead of a function. IMHO
 unless/until major architectural changes to the GC are made that require
 a function pointer, there's no point in adding this indirection.

 I started working on this and I ran into a roadblock. I need to know
 what parts of the GC API are allowed to change, and discuss how to
 abstract away the implementation of it from the GC API. I assume the
 stuff in core.memory needs to stay mostly the same, though I guess we
 would need to add a setType() function that takes a pointer into a block
 of memory and a TypeInfo object and changes how the GC interprets the
 bits in the block.

 In gc.d, we define a bunch of extern(C) functions and the proxy thing.
 Since we've given up on the idea of swapping precise GCs at link time,
 can I just rip out all this unnecesary indirection? If not, is it ok to
 change some of these signatures? I definitely want to avoid allocating
 (requiring the GC lock) and then calling a function to set the type
 (requiring another lock acquisition) so the signature of malloc(), etc.
 needs to change somewhere.

 More generally, what is the intended way to get GCInfo pointers from
 TypeInfo into the guts of the GC where they can be acted on?

This is not specifically an answer to your question, but my opinion on this subject is that altering the GC API without a deprecation process or similar is fine. This is a core component of the runtime, and I don't think expecting a stable API for something like a garbage collector of all things is reasonable in the first place. That said, I do think functionality has to be maintained. That is, if the GC can do something in a previous version, it should be able to do it in the subsequent version too. *How* that something is achieved in the subsequent version isn't so important, as long as the functionality is *there* in some way. -- - Alex
Apr 18 2012
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/17/2012 5:36 PM, dsimcha wrote:
 Now that the compiler infrastructure has been implemented, I've gotten busy
 figuring out how to make D's default GC precise. As a first attempt, I think
I'm
 going to adapt my original solution from
 http://d.puremagic.com/issues/show_bug.cgi?id=3463 since it's simple and it
 works except that there previously was no clean way to get the offset info into
 the GC. As Walter pointed out in another thread, the GCInfo template is allowed
 to instantiate to data instead of a function. IMHO unless/until major
 architectural changes to the GC are made that require a function pointer,
 there's no point in adding this indirection.

 I started working on this and I ran into a roadblock. I need to know what parts
 of the GC API are allowed to change, and discuss how to abstract away the
 implementation of it from the GC API. I assume the stuff in core.memory needs
to
 stay mostly the same, though I guess we would need to add a setType() function
 that takes a pointer into a block of memory and a TypeInfo object and changes
 how the GC interprets the bits in the block.

 In gc.d, we define a bunch of extern(C) functions and the proxy thing. Since
 we've given up on the idea of swapping precise GCs at link time, can I just rip
 out all this unnecesary indirection? If not, is it ok to change some of these
 signatures? I definitely want to avoid allocating (requiring the GC lock) and
 then calling a function to set the type (requiring another lock acquisition) so
 the signature of malloc(), etc. needs to change somewhere.

 More generally, what is the intended way to get GCInfo pointers from TypeInfo
 into the guts of the GC where they can be acted on?

1. I would not try to redesign everything and do precise gc at the same time. 2. The purpose of the indirection is to support DLLs so that the different DLLs can share an instance. 3. The reason for function pointers for marking is so that the marking code can be customized and directly inlined, rather than decoding a table. It costs one code indirection, but after that it cannot be beaten for speed.
Apr 18 2012
parent dsimcha <dsimcha yahoo.com> writes:
On 4/18/2012 6:46 PM, Sean Kelly wrote:
 Leandro's GC (CDGC) is already set up to support precise scanning.  It's in
the Druntime git repository, but lacks the features added to the Druntime GC
compared to the Tango GC on which CDGC is based.  Still, it may be easier to
update CDGC based on a diff between the Druntime and Tango GC than it would to
add precise scanning to the GC Druntime currently uses.  Worth a look if anyone
is interested anyway.

Or, failing that, I can look at it to get ideas about how to handle various annoying plumbing issues. The plumbing issues (i.e. getting the GCInfo pointers from the allocation routines into the guts of the GC) are actually the hard part of this project. Once the GC has the the GCInfo pointer, making it use that for precise scanning is trivial in that I've done it before and remember roughly how I did it.
Apr 18 2012
prev sibling next sibling parent Sean Kelly <sean invisibleduck.org> writes:
On Apr 18, 2012, at 2:56 AM, deadalnix wrote:
=20
 I think you certainly can remove all indirection. Additionally, I =

So the GC implementation is opaque and the GC can therefore be chosen at = link-time. Similar to how the compiler runtime code hides behind a raft = of extern=A9 functions.=
Apr 18 2012
prev sibling next sibling parent Sean Kelly <sean invisibleduck.org> writes:
On Apr 17, 2012, at 5:36 PM, dsimcha wrote:
=20
 More generally, what is the intended way to get GCInfo pointers from =

I just resurrected an old thread titled "Proposed changes to the GC = interface" in the Druntime mailing list. Perhaps we could pick up = discussion there?
Apr 18 2012
prev sibling next sibling parent Sean Kelly <sean invisibleduck.org> writes:
On Apr 18, 2012, at 1:13 PM, Walter Bright wrote:

 On 4/17/2012 5:36 PM, dsimcha wrote:
 Now that the compiler infrastructure has been implemented, I've =


 figuring out how to make D's default GC precise. As a first attempt, =


 going to adapt my original solution from
 http://d.puremagic.com/issues/show_bug.cgi?id=3D3463 since it's =


 works except that there previously was no clean way to get the offset =


 the GC. As Walter pointed out in another thread, the GCInfo template =


 to instantiate to data instead of a function. IMHO unless/until major
 architectural changes to the GC are made that require a function =


 there's no point in adding this indirection.
=20
 I started working on this and I ran into a roadblock. I need to know =


 of the GC API are allowed to change, and discuss how to abstract away =


 implementation of it from the GC API. I assume the stuff in =


 stay mostly the same, though I guess we would need to add a setType() =


 that takes a pointer into a block of memory and a TypeInfo object and =


 how the GC interprets the bits in the block.
=20
 In gc.d, we define a bunch of extern(C) functions and the proxy =


 we've given up on the idea of swapping precise GCs at link time, can =


 out all this unnecesary indirection? If not, is it ok to change some =


 signatures? I definitely want to avoid allocating (requiring the GC =


 then calling a function to set the type (requiring another lock =


 the signature of malloc(), etc. needs to change somewhere.
=20
 More generally, what is the intended way to get GCInfo pointers from =


 into the guts of the GC where they can be acted on?

1. I would not try to redesign everything and do precise gc at the =

=20
 2. The purpose of the indirection is to support DLLs so that the =

=20
 3. The reason for function pointers for marking is so that the marking =

table. It costs one code indirection, but after that it cannot be beaten = for speed. Leandro's GC (CDGC) is already set up to support precise scanning. It's = in the Druntime git repository, but lacks the features added to the = Druntime GC compared to the Tango GC on which CDGC is based. Still, it = may be easier to update CDGC based on a diff between the Druntime and = Tango GC than it would to add precise scanning to the GC Druntime = currently uses. Worth a look if anyone is interested anyway.
Apr 18 2012
prev sibling parent Sean Kelly <sean invisibleduck.org> writes:
On Apr 18, 2012, at 4:02 PM, deadalnix wrote:

 Le 18/04/2012 20:53, Sean Kelly a =E9crit :
 On Apr 18, 2012, at 2:56 AM, deadalnix wrote:
=20
 I think you certainly can remove all indirection. Additionally, I =



=20
 So the GC implementation is opaque and the GC can therefore be chosen =


raft of extern=A9 functions.
=20
 I know, but this is now impossible anyway because of the modification =

I'm not sure I follow. Are you saying the change breaks having the GC = behind extern C functions? How?
Apr 18 2012