www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - GC, memory leaks and 32/64 bit

reply Alexandr Druzhinin <drug2004 bk.ru> writes:
32 bit versions (using dmd and gdc, win7/ubuntu 12.04) of my application 
always leak very fast. But 64 bit version (ubuntu 12.04 only, win7 
segfaults) works stable without any leaks. I'm curious is there some 
workaround or I have to wait for GC improvement to build 32bit version, 
without choice?
Mar 12 2013
parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Alexandr Druzhinin:


 32 bit versions (using dmd and gdc, win7/ubuntu 12.04) of my 
 application always leak very fast. But 64 bit version (ubuntu 
 12.04 only, win7 segfaults) works stable without any leaks. I'm 
 curious is there some workaround or I have to wait for GC 
 improvement to build 32bit version, without choice?


This difference is caused by the difference in pointer space size 
coupled with the nature of a conservative GC.
The "workarounds" are not easy, like allocating your largest 
arrays from the C heap.
The past Summer Of Code was worked on a much more precise GC (but 
not fully precise) meant to reduce this problem a lot on 32 bit 
systems, but nothing solid has so far come out of that. It 
probably needs to be polished, tested, etc.

Bye,
bearophile
Mar 12 2013
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 12.03.2013 14:07, schrieb bearophile:

 Alexandr Druzhinin:


 32 bit versions (using dmd and gdc, win7/ubuntu 12.04) of my
 application always leak very fast. But 64 bit version (ubuntu 12.04
 only, win7 segfaults) works stable without any leaks. I'm curious is
 there some workaround or I have to wait for GC improvement to build
 32bit version, without choice?


 This difference is caused by the difference in pointer space size
 coupled with the nature of a conservative GC.
 The "workarounds" are not easy, like allocating your largest arrays from
 the C heap.
 The past Summer Of Code was worked on a much more precise GC (but not
 fully precise) meant to reduce this problem a lot on 32 bit systems, but
 nothing solid has so far come out of that. It probably needs to be
 polished, tested, etc.

 Bye,
 bearophile


Thats not correct. Rainer Schuetze has finished it and is using it for 
VisualD. You can get a version of druntime which the percise GC from his 
github branch https://github.com/rainers/dmd

Kind Regards
Benjamin Thaut
Mar 12 2013
next sibling parent Benjamin Thaut <code benjamin-thaut.de> writes:
Ah sorry, I meant to post: 
https://github.com/rainers/druntime/tree/precise_gc2

Kind Regards
Benjamin Thaut
Mar 12 2013
prev sibling next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Benjamin Thaut:


 Thats not correct. Rainer Schuetze has finished it and is using 
 it for VisualD. You can get a version of druntime which the 
 percise GC from his github branch https://github.com/rainers/dmd


I am glad to be wrong :-)

Do you know how well the new GC is working?

Bye,
bearophile
Mar 12 2013
next sibling parent "simendsjo" <simendsjo gmail.com> writes:
On Tuesday, 12 March 2013 at 14:21:06 UTC, bearophile wrote:

 Benjamin Thaut:


 Thats not correct. Rainer Schuetze has finished it and is 
 using it for VisualD. You can get a version of druntime which 
 the percise GC from his github branch 
 https://github.com/rainers/dmd


 I am glad to be wrong :-)

 Do you know how well the new GC is working?


Haven't sociomatic (Don) used it in production a while too? I 
think I remember reading something on that, and that it was 
working good.
Mar 12 2013
prev sibling next sibling parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 12.03.2013 15:21, schrieb bearophile:

 Benjamin Thaut:


 Thats not correct. Rainer Schuetze has finished it and is using it for
 VisualD. You can get a version of druntime which the percise GC from
 his github branch https://github.com/rainers/dmd


 I am glad to be wrong :-)

 Do you know how well the new GC is working?

 Bye,
 bearophile


Rainer stated that it works just fine, but in its current state it is 
slower then the impercicse GC.

Kind Regards
Benjamin Thaut
Mar 12 2013
prev sibling parent reply "Rob T" <alanb ucora.com> writes:
On Tuesday, 12 March 2013 at 14:21:06 UTC, bearophile wrote:

 Benjamin Thaut:


 Thats not correct. Rainer Schuetze has finished it and is 
 using it for VisualD. You can get a version of druntime which 
 the percise GC from his github branch 
 https://github.com/rainers/dmd


 I am glad to be wrong :-)

 Do you know how well the new GC is working?

 Bye,
 bearophile


It should be a plugin so that components like an alternate GC can 
be swapped in and out easily. I hope once shared lib support 
becomes available we'll finally see plugins introduced into dmd.

--rt
Mar 12 2013
next sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 12 Mar 2013 21:08:01 +0100
schrieb "Rob T" <alanb ucora.com>:


 It should be a plugin so that components like an alternate GC can 
 be swapped in and out easily. I hope once shared lib support 
 becomes available we'll finally see plugins introduced into dmd.
 
 --rt


Well, you can already today set a proxy for the GC. This
mainly to disable a DLLs GC and make it foward everything to
the main executable's GC, but you can use for other purposes.
It just means an additional indirect function call for every
GC method call.
An alternative is to override all of the exported GC C
functions with your own implementations, so that the linker
will prefer the versions found in your code rather than the
ones in Phobos. But this doesn't allow swapping at runtime.

-- 
Marco
Mar 12 2013
parent reply "Rob T" <alanb ucora.com> writes:
On Tuesday, 12 March 2013 at 20:29:58 UTC, Marco Leise wrote:

 Am Tue, 12 Mar 2013 21:08:01 +0100
 schrieb "Rob T" <alanb ucora.com>:


 It should be a plugin so that components like an alternate GC 
 can be swapped in and out easily. I hope once shared lib 
 support becomes available we'll finally see plugins introduced 
 into dmd.
 
 --rt


 Well, you can already today set a proxy for the GC. This
 mainly to disable a DLLs GC and make it foward everything to
 the main executable's GC, but you can use for other purposes.
 It just means an additional indirect function call for every
 GC method call.
 An alternative is to override all of the exported GC C
 functions with your own implementations, so that the linker
 will prefer the versions found in your code rather than the
 ones in Phobos. But this doesn't allow swapping at runtime.


I don't think you'd want to swap a GC during runtime anyway, but 
who knows, so having that ability is probably better than not 
having it.

Do you know if there are instructions posted somewhere on how to 
override or proxy the GC?

--rt
Mar 12 2013
parent Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 12 Mar 2013 21:44:11 +0100
schrieb "Rob T" <alanb ucora.com>:


 Do you know if there are instructions posted somewhere on how to 
 override or proxy the GC?
 
 --rt


Maybe someone else knows, I just started from here:

https://github.com/D-Programming-Language/druntime/blob/master/src/gcstub/gc.d

When you look at the "real" GC you'll quickly find the C
funtions that you'd need to override.


-- 
Marco
Mar 12 2013
prev sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 12.03.2013 21:08, schrieb Rob T:

 On Tuesday, 12 March 2013 at 14:21:06 UTC, bearophile wrote:

 Benjamin Thaut:


 Thats not correct. Rainer Schuetze has finished it and is using it
 for VisualD. You can get a version of druntime which the percise GC
 from his github branch https://github.com/rainers/dmd


 I am glad to be wrong :-)

 Do you know how well the new GC is working?

 Bye,
 bearophile


 It should be a plugin so that components like an alternate GC can be
 swapped in and out easily. I hope once shared lib support becomes
 available we'll finally see plugins introduced into dmd.

 --rt


This is not possible as different kinds of GCs need to generate 
different runtime data at compile time. The current GC for example does 
not need any runtime data (other then what the D typeinfo system already 
provides) but the percicse GC needs additional runtime information, so 
you can't just "swap" them. You need to recompile.

Kind Regards
Benjamin Thaut
Mar 12 2013
next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Benjamin Thaut:


 so you can't just "swap" them. You need to recompile.


Is it a good idea to add a compiler switch to specify what GC to 
use?

Bye,
bearophile
Mar 12 2013
prev sibling parent reply "Rob T" <alanb ucora.com> writes:
On Tuesday, 12 March 2013 at 21:02:20 UTC, Benjamin Thaut wrote:

 This is not possible as different kinds of GCs need to generate 
 different runtime data at compile time. The current GC for 
 example does not need any runtime data (other then what the D 
 typeinfo system already provides) but the percicse GC needs 
 additional runtime information, so you can't just "swap" them. 
 You need to recompile.

 Kind Regards
 Benjamin Thaut


I don't know much about the problem, so this may be a dumb 
question, but I'll ask anyway.

Can a loadable GC be made possible with an improved design? For 
example, can the component that generates the needed runtime data 
for the GC also be plugged in?

--rt
Mar 12 2013
parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 13.03.2013 06:38, schrieb Rob T:

 On Tuesday, 12 March 2013 at 21:02:20 UTC, Benjamin Thaut wrote:

 This is not possible as different kinds of GCs need to generate
 different runtime data at compile time. The current GC for example
 does not need any runtime data (other then what the D typeinfo system
 already provides) but the percicse GC needs additional runtime
 information, so you can't just "swap" them. You need to recompile.

 Kind Regards
 Benjamin Thaut


 I don't know much about the problem, so this may be a dumb question, but
 I'll ask anyway.

 Can a loadable GC be made possible with an improved design? For example,
 can the component that generates the needed runtime data for the GC also
 be plugged in?

 --rt


The GC could load the needed runtime data from a file. But it would have 
to be generated a compile time anyway. so you always need to know at 
compile time which GCs you migth want to use in the future.

-- 
Kind Regards
Benjamin Thaut
Mar 13 2013
prev sibling parent reply Druzhinin Alexandr <news digitalmars.com> writes:
On 12.03.2013 20:23, Benjamin Thaut wrote:

 Am 12.03.2013 14:07, schrieb bearophile:

 Alexandr Druzhinin:


 32 bit versions (using dmd and gdc, win7/ubuntu 12.04) of my
 application always leak very fast. But 64 bit version (ubuntu 12.04
 only, win7 segfaults) works stable without any leaks. I'm curious is
 there some workaround or I have to wait for GC improvement to build
 32bit version, without choice?


 This difference is caused by the difference in pointer space size
 coupled with the nature of a conservative GC.
 The "workarounds" are not easy, like allocating your largest arrays from
 the C heap.
 The past Summer Of Code was worked on a much more precise GC (but not
 fully precise) meant to reduce this problem a lot on 32 bit systems, but
 nothing solid has so far come out of that. It probably needs to be
 polished, tested, etc.

 Bye,
 bearophile


 Thats not correct. Rainer Schuetze has finished it and is using it for
 VisualD. You can get a version of druntime which the percise GC from his
 github branch https://github.com/rainers/dmd

 Kind Regards
 Benjamin Thaut


I tried to build dmd 2.062 + Rainer's druntime + phobos 2.062 + some 
code manipulation to make it compiles but I got undefined symbols (BTW 
may I link it with druntime 2.062 (besides Rainer's one) to get these 
symbols?).
I guess it would be better to use 2.061 but my application doesn't 
compile with it now, will try it later. And it would be very nice to 
have some instructions about using alternative gc.
Mar 12 2013
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 13.03.2013 06:46, Druzhinin Alexandr wrote:

 On 12.03.2013 20:23, Benjamin Thaut wrote:

 Am 12.03.2013 14:07, schrieb bearophile:

 Alexandr Druzhinin:


 32 bit versions (using dmd and gdc, win7/ubuntu 12.04) of my
 application always leak very fast. But 64 bit version (ubuntu 12.04
 only, win7 segfaults) works stable without any leaks. I'm curious is
 there some workaround or I have to wait for GC improvement to build
 32bit version, without choice?


 This difference is caused by the difference in pointer space size
 coupled with the nature of a conservative GC.
 The "workarounds" are not easy, like allocating your largest arrays from
 the C heap.
 The past Summer Of Code was worked on a much more precise GC (but not
 fully precise) meant to reduce this problem a lot on 32 bit systems, but
 nothing solid has so far come out of that. It probably needs to be
 polished, tested, etc.

 Bye,
 bearophile


 Thats not correct. Rainer Schuetze has finished it and is using it for
 VisualD. You can get a version of druntime which the percise GC from his
 github branch https://github.com/rainers/dmd

 Kind Regards
 Benjamin Thaut


 I tried to build dmd 2.062 + Rainer's druntime + phobos 2.062 + some
 code manipulation to make it compiles but I got undefined symbols (BTW
 may I link it with druntime 2.062 (besides Rainer's one) to get these
 symbols?).
 I guess it would be better to use 2.061 but my application doesn't
 compile with it now, will try it later. And it would be very nice to
 have some instructions about using alternative gc.


If the undefined symbols happen to be related to AssociativeArray, my 
current workaround is to add "alias Associative!(Key,Value) 
_workaround;" somewhere to force instantiation for the respective Key 
and Value types.

The precise GC needs some type info for the allocated memory, so I have 
added a TypeInfo parameter to most calls into the GC. This changes the 
interface, so the precise cannot be plugged into the current proxy 
implementation.

I'm currently cleaning up the precise GC implementation so it can be 
switched on with a version when compiling druntime. Here is the current 
state: https://github.com/rainers/druntime/tree/gcx_precise
Mar 12 2013
parent reply Druzhinin Alexandr <news digitalmars.com> writes:
On 13.03.2013 13:35, Rainer Schuetze wrote:

 If the undefined symbols happen to be related to AssociativeArray, my
 current workaround is to add "alias Associative!(Key,Value)
 _workaround;" somewhere to force instantiation for the respective Key
 and Value types.

 The precise GC needs some type info for the allocated memory, so I have
 added a TypeInfo parameter to most calls into the GC. This changes the
 interface, so the precise cannot be plugged into the current proxy
 implementation.

 I'm currently cleaning up the precise GC implementation so it can be
 switched on with a version when compiling druntime. Here is the current
 state: https://github.com/rainers/druntime/tree/gcx_precise


What version of dmd and phobos I should use to link against your 
druntime? Undefined symbols weren't AA, its were core.memory and 
something else, but they I guess were the result of different versions 
phobos and druntime.
Mar 13 2013
parent Rainer Schuetze <r.sagitario gmx.de> writes:
On 13.03.2013 08:03, Druzhinin Alexandr wrote:

 On 13.03.2013 13:35, Rainer Schuetze wrote:

 If the undefined symbols happen to be related to AssociativeArray, my
 current workaround is to add "alias Associative!(Key,Value)
 _workaround;" somewhere to force instantiation for the respective Key
 and Value types.

 The precise GC needs some type info for the allocated memory, so I have
 added a TypeInfo parameter to most calls into the GC. This changes the
 interface, so the precise cannot be plugged into the current proxy
 implementation.

 I'm currently cleaning up the precise GC implementation so it can be
 switched on with a version when compiling druntime. Here is the current
 state: https://github.com/rainers/druntime/tree/gcx_precise


 What version of dmd and phobos I should use to link against your
 druntime? Undefined symbols weren't AA, its were core.memory and
 something else, but they I guess were the result of different versions
 phobos and druntime.


There were some recent changes with respect to inout on TypeInfo.next 
that don't work with older compilers, so I guess it will only work with 
a pretty recent version of dmd from github. Regarding phobos, it should 
be built on top of druntime from the branch.
Mar 13 2013