www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - D wrapper classes leak memory. Next steps?

reply Marco Leise <Marco.Leise gmx.de> writes:
GtkD wraps all the objects in Gtk. In callbacks like "onDraw",
when they are called often, this creates heaps of tiny wrapper
objects around huge data from C libraries. Eventually system
memory is exhausted and the computer slows down and eats into
swap. So should we add a "run garbage collector" button to
every application to clean those up? Seems rather silly.

The GC should stick to its own memory and not try to manage
resources it doesn't understand. Reference counting applies to
external and system resources much better. To date neither the
language nor Phobos can ref count classes. And even with
library support it is not likely that it will get used widely,
because plain classes are so much easier (also on the eyes).

The extent of this is pretty huge. There are many D wrapper
libraries that build on classes and mimic what is available
for C++ or Python. Issues like this make D appear like a toy
language. How would one write a big desktop application when
a simple animation can exhaust system memory in seconds ?

-- 
Marco
Oct 30 2014
next sibling parent reply ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Fri, 31 Oct 2014 05:38:48 +0100
Marco Leise via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 GtkD wraps all the objects in Gtk. In callbacks like "onDraw",
 when they are called often, this creates heaps of tiny wrapper
 objects around huge data from C libraries. Eventually system
 memory is exhausted and the computer slows down and eats into
 swap. So should we add a "run garbage collector" button to
 every application to clean those up? Seems rather silly.
=20
 The GC should stick to its own memory and not try to manage
 resources it doesn't understand. Reference counting applies to
 external and system resources much better. To date neither the
 language nor Phobos can ref count classes. And even with
 library support it is not likely that it will get used widely,
 because plain classes are so much easier (also on the eyes).
=20
 The extent of this is pretty huge. There are many D wrapper
 libraries that build on classes and mimic what is available
 for C++ or Python. Issues like this make D appear like a toy
 language. How would one write a big desktop application when
 a simple animation can exhaust system memory in seconds ?
why do you blaming GC for what seems to be a library author's fault? that was the library author who chooses to wrap externally-managed objects in GC objects, and creating alot of throwaway classes for that. alas, we haven't Skynet at the hand to guess what author really wants. p.s. i was never using GtkD, so i don't know if it's really doing the things you wrote or not, i'm just assuming that you are right.
Oct 30 2014
parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Fri, 31 Oct 2014 06:44:48 +0200
schrieb ketmar via Digitalmars-d <digitalmars-d puremagic.com>:

 On Fri, 31 Oct 2014 05:38:48 +0100
 Marco Leise via Digitalmars-d <digitalmars-d puremagic.com> wrote:
 
 GtkD wraps all the objects in Gtk. In callbacks like "onDraw",
 when they are called often, this creates heaps of tiny wrapper
 objects around huge data from C libraries. Eventually system
 memory is exhausted and the computer slows down and eats into
 swap. So should we add a "run garbage collector" button to
 every application to clean those up? Seems rather silly.
 
 The GC should stick to its own memory and not try to manage
 resources it doesn't understand. Reference counting applies to
 external and system resources much better. To date neither the
 language nor Phobos can ref count classes. And even with
 library support it is not likely that it will get used widely,
 because plain classes are so much easier (also on the eyes).
 
 The extent of this is pretty huge. There are many D wrapper
 libraries that build on classes and mimic what is available
 for C++ or Python. Issues like this make D appear like a toy
 language. How would one write a big desktop application when
 a simple animation can exhaust system memory in seconds ?
why do you blaming GC for what seems to be a library author's fault? that was the library author who chooses to wrap externally-managed objects in GC objects, and creating alot of throwaway classes for that.
Can't blame the author when there is no other choice in the language than to use tracing GC with inheritance.
 alas, we haven't Skynet at the hand to guess what author really wants.
 
 p.s. i was never using GtkD, so i don't know if it's really doing the
 things you wrote or not, i'm just assuming that you are right.
To be fair there is an overload for addOnDraw that takes a delegate that has a Scoped!Context as parameter, but I cannot use that with delegates marked as nothrow since Scoped's dtor isn't nothrow. Which prompts the question again how to deal with exceptions in dtors. All D delegates called from the C side have to be nothrow though, because neither can druntime deal with GCC's amd64 stack (omit-frame-pointer) nor is the C side necessarily prepared for exceptions. -- Marco
Oct 31 2014
parent reply ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Fri, 31 Oct 2014 09:46:54 +0100
Marco Leise via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 Can't blame the author when there is no other choice in the
 language than to use tracing GC with inheritance.
there are alot of choices. i'm succesfully using wrapper classes with reference counting in my i/o streams library, for example. they are GC roots, and they are freed in the same moment when their reference count becomes zero.
 Which prompts the question again how to deal
 with exceptions in dtors.
class dtors should not throw any exceptions. stack-allocated struct dtors can do what they want, just catch that and you'll got "nothrow" function. what's the problem here?
Oct 31 2014
parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Fri, 31 Oct 2014 10:51:04 +0200
schrieb ketmar via Digitalmars-d <digitalmars-d puremagic.com>:

 On Fri, 31 Oct 2014 09:46:54 +0100
 Marco Leise via Digitalmars-d <digitalmars-d puremagic.com> wrote:
 
 Can't blame the author when there is no other choice in the
 language than to use tracing GC with inheritance.
there are alot of choices. i'm succesfully using wrapper classes with reference counting in my i/o streams library, for example. they are GC roots, and they are freed in the same moment when their reference count becomes zero.
I meant obvious, clean, idiomatic choices. It's clear that D's expressiveness makes pretty much anything work if you put a week's worth of time into it. What I'm saying is, the language should offer something to use external resources in class hierarchies, because it is a common need. Something that comes with so little friction that you don't think twice.
 Which prompts the question again how to deal
 with exceptions in dtors.
class dtors should not throw any exceptions. stack-allocated struct dtors can do what they want, just catch that and you'll got "nothrow" function. what's the problem here?
E-he, stack sounds nice, but those are in D's "parameter scope". You cannot wrap their destruction in try-catch. -- Marco
Oct 31 2014
parent reply ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Fri, 31 Oct 2014 10:53:27 +0100
Marco Leise via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 Am Fri, 31 Oct 2014 10:51:04 +0200
 schrieb ketmar via Digitalmars-d <digitalmars-d puremagic.com>:
=20
 On Fri, 31 Oct 2014 09:46:54 +0100
 Marco Leise via Digitalmars-d <digitalmars-d puremagic.com> wrote:
=20
 Can't blame the author when there is no other choice in the
 language than to use tracing GC with inheritance.
there are alot of choices. i'm succesfully using wrapper classes with reference counting in my i/o streams library, for example. they are GC roots, and they are freed in the same moment when their reference count becomes zero.
=20 I meant obvious, clean, idiomatic choices. It's clear that D's expressiveness makes pretty much anything work if you put a week's worth of time into it. What I'm saying is, the language should offer something to use external resources in class hierarchies, because it is a common need. Something that comes with so little friction that you don't think twice.
if you have something concrete in mind, write ER or forum post, so we can destroy it. ;-)
 Which prompts the question again how to deal
 with exceptions in dtors.
class dtors should not throw any exceptions. stack-allocated struct dtors can do what they want, just catch that and you'll got "nothrow" function. what's the problem here?
=20 E-he, stack sounds nice, but those are in D's "parameter scope". You cannot wrap their destruction in try-catch.
there is no problem that can't be solved by introducing one more wrapper. ;-)
Oct 31 2014
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 31 October 2014 at 09:58:41 UTC, ketmar via 
Digitalmars-d wrote:
 if you have something concrete in mind, write ER or forum post, 
 so we can destroy it. ;-)
Sure! - dump the gc - make uniqe_ptr a language builtin - ditch c malloc - make compiler 100% memory management aware - make thread aware high speed class allocators a builtin - ditch emplace - default pointers to non-nullable - do a complete redesign of phobos I'm sure there is more…
Oct 31 2014
next sibling parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Fri, 31 Oct 2014 10:06:50 +0000
via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 On Friday, 31 October 2014 at 09:58:41 UTC, ketmar via=20
 Digitalmars-d wrote:
 if you have something concrete in mind, write ER or forum post,=20
 so we can destroy it. ;-)
=20 Sure! =20 - dump the gc =20 - make uniqe_ptr a language builtin =20 - ditch c malloc =20 - make compiler 100% memory management aware =20 - make thread aware high speed class allocators a builtin =20 - ditch emplace =20 - default pointers to non-nullable =20 - do a complete redesign of phobos =20 I'm sure there is more=E2=80=A6
send your patches! ;-)
Oct 31 2014
prev sibling next sibling parent reply ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Fri, 31 Oct 2014 10:06:50 +0000
via Digitalmars-d <digitalmars-d puremagic.com> wrote:

p.s. and title that "Z", as "the last language you'll need ever. in all
meanings." ;-)
Oct 31 2014
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 31 October 2014 at 10:21:47 UTC, ketmar via 
Digitalmars-d wrote:
 On Fri, 31 Oct 2014 10:06:50 +0000
 via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 p.s. and title that "Z", as "the last language you'll need 
 ever. in all meanings." ;-)
Indeed, for the english speaking world, but as a patriotic scandinavian I'm inclined to think there are three more letters. And I cannot have that, can I? In Denmark and Norway the alphabet ends with : Z, Æ, Ø, Å… And in Sweden: Z, Å, Ä, Ö… So the name of the ultimate language has to be one of: ZÅÖ ZÖÅ ÅZÖ ÅÖZ ÖZÅ or ÖÅZ Hm…
Oct 31 2014
next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 31 October 2014 at 12:09:29 UTC, Ola Fosheim Grøstad 
wrote:
 ZÅÖ
 ZÖÅ
 ÅZÖ
 ÅÖZ
 ÖZÅ
 or
 ÖÅZ

 Hm…
On second thought I think I should stick to the Norwegian alphabet, but resort to old spelling rules. That will make the name of the ultimate language: "AA", pronounced "Awe": http://www.youtube.com/watch?v=BX5W8gXcAW4 And, as a bonus it will be listed first in programming language listings, such as Rosetta Code. OK. That's it. I am from now on working on a new system level programming language called "AA", pronounced "Awe" which should be sorted first in all english listings, and sorted last in all scandinavian listings. No more patches. I'm speccing!
Oct 31 2014
prev sibling parent reply ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Fri, 31 Oct 2014 12:09:27 +0000
via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 On Friday, 31 October 2014 at 10:21:47 UTC, ketmar via=20
 Digitalmars-d wrote:
 On Fri, 31 Oct 2014 10:06:50 +0000
 via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 p.s. and title that "Z", as "the last language you'll need=20
 ever. in all meanings." ;-)
=20 Indeed, for the english speaking world, but as a patriotic=20 scandinavian I'm inclined to think there are three more letters.=20 And I cannot have that, can I? =20 In Denmark and Norway the alphabet ends with : Z, =C3=86, =C3=98, =C3=85=
=E2=80=A6
 And in Sweden: Z, =C3=85, =C3=84, =C3=96=E2=80=A6
=20
 So the name of the ultimate language has to be one of:
=20
 Z=C3=85=C3=96
 Z=C3=96=C3=85
 =C3=85Z=C3=96
 =C3=85=C3=96Z
 =C3=96Z=C3=85
 or
 =C3=96=C3=85Z
=20
 Hm=E2=80=A6
i bet you can find a nicely looking hieroglyph too. that would be cool for graffiti.
Oct 31 2014
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 31 October 2014 at 12:22:56 UTC, ketmar via 
Digitalmars-d wrote:
 i bet you can find a nicely looking hieroglyph too. that would 
 be cool for graffiti.
Good point. So the logo would be "Å", it is spelled "AA", pronounced "Awe" and sorted both first and last in listings. It has to be a very minimalistic language: http://en.wikipedia.org/wiki/Angstrom
Oct 31 2014
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 31 October 2014 at 12:31:25 UTC, Ola Fosheim Grøstad 
wrote:
 On Friday, 31 October 2014 at 12:22:56 UTC, ketmar via 
 Digitalmars-d wrote:
 i bet you can find a nicely looking hieroglyph too. that would 
 be cool for graffiti.
Good point. So the logo would be "Å", it is spelled "AA", pronounced "Awe" and sorted both first and last in listings. It has to be a very minimalistic language: http://en.wikipedia.org/wiki/Angstrom
Oh drat, someone is squatting my would-be website… http://www.aa.org/
Oct 31 2014
parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Fri, 31 Oct 2014 12:56:58 +0000
via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 Oh drat, someone is squatting my would-be website=E2=80=A6
 http://www.aa.org/
ah. just wait until they got drunk and buy it in exchange for one more shot.
Oct 31 2014
prev sibling parent reply "eles" <eles eles.com> writes:
On Friday, 31 October 2014 at 10:06:51 UTC, Ola Fosheim Grøstad 
wrote:
 On Friday, 31 October 2014 at 09:58:41 UTC, ketmar via 
 Digitalmars-d wrote:
 if you have something concrete in mind, write ER or forum 
 post, so we can destroy it. ;-)
 - make uniqe_ptr a language builtin
I would go with this by default for all pointers/references. Something that I read here: https://news.ycombinator.com/item?id=7650917 My feeling too. If you need more owners for a resource, then explicitly ask for.
Oct 31 2014
next sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 31 October 2014 at 13:16:17 UTC, eles wrote:
 Something that I read here:

 https://news.ycombinator.com/item?id=7650917

 My feeling too.

 If you need more owners for a resource, then explicitly ask for.
Thank you for sharing, a very interesting discussion topic. I am increasingly thinking that it might be possible to have a system level language where you can do progressive refinement from a garbage collection supported sketch to a hardcore real time app. Basically have different compilation modes where you start out in executable pseudo-code and gradually turn on more switches which give you performance/ownership etc warnings (integrated in the IDE). Kinda a like starting out with Python and ending up with C. D supports this a little bit, but has not been designed for it (I think).
Oct 31 2014
parent reply "eles" <eles215 gzk.dot> writes:
On Friday, 31 October 2014 at 13:42:53 UTC, Ola Fosheim Grøstad 
wrote:
 On Friday, 31 October 2014 at 13:16:17 UTC, eles wrote:
 Something that I read here:

 https://news.ycombinator.com/item?id=7650917

 My feeling too.

 If you need more owners for a resource, then explicitly ask 
 for.
Thank you for sharing, a very interesting discussion topic.
Just skimming over this: http://dlang.org/const-faq.html#const-parameters it would seem to make sense to make them all const by default, and one would have to specifically mark as mutable those that would be changed. The problems with this are: It would be a huge break from past D practice, and practice in C, C++, Java, C#, etc. -- Actually, this is the sole reason, but the breakage would be far less that the breakage introduced by Rust & co. The two reasons that follow are almost hypocrisy to justify the fear in point 1. It would require a new keyword, say mutable. -- Big deal. C++ has mutable too. And worst, it would make declarations inconsistent: void foo(int* p) { int* q; ... } p points to const, and q points to mutable. This kind of inconsistency leads to all sorts of mistakes. It also makes it very hard to write generic code that deals with types. -- Why not make all pointer declarations as constant by default? Then, you explicitely ask for a mutable reference (hell, you could even limit the number of mutable references that could be asked for, if you want). Just reformulate to say: "p points to const, and q points to const. This kind of consistency leads to all kind of great code. It also makes it very easy to write generic code that deals with types."
Oct 31 2014
parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 31 October 2014 at 21:43:26 UTC, eles wrote:
 it would seem to make sense to make them all const by default, 
 and one would have to specifically mark as mutable those that 
 would be changed.
Yes, I think that would be interesting. I also thought about the possibility of putting all mutable parameters to the left of a dot-operator, but it takes some time getting used to (not sure if I could get used to it): (arr1,arr2).swap_range(10,20); arr1.merge(arr2); //merge by copy (arr1,arr2).merge; // merge by moving
 -- Why not make all pointer declarations as constant by 
 default? Then, you explicitely ask for a mutable reference 
 (hell, you could even limit the number of mutable references 
 that could be asked for, if you want). Just reformulate to say:
I think immutable/const by default is the best option. It is primarily in loops that mutable is needed and since mutables are more likely to cause bugs it would be nice to have some attention drawn to them.
Oct 31 2014
prev sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Fri, 31 Oct 2014 13:16:15 +0000
schrieb "eles" <eles eles.com>:

 On Friday, 31 October 2014 at 10:06:51 UTC, Ola Fosheim Gr=C3=B8stad=20
 wrote:
 On Friday, 31 October 2014 at 09:58:41 UTC, ketmar via=20
 Digitalmars-d wrote:
 if you have something concrete in mind, write ER or forum=20
 post, so we can destroy it. ;-)
=20
 - make uniqe_ptr a language builtin
=20 I would go with this by default for all pointers/references. =20 Something that I read here: =20 https://news.ycombinator.com/item?id=3D7650917 =20 My feeling too. =20 If you need more owners for a resource, then explicitly ask for.
In case of a onDraw callback the object will be passed as a parameter into the drawing function. unique_ptr doesn't solve this. In the particular case of callbacks from C libraries that give us opaque "context" pointers we want to wrap with D classes we can use either of: - reference counted object (doesn't exist in D) - unique_ptr (requires "scope" to be implemented) - Scoped ? Maybe Scoped wasn't explored enough for callbacks in GtkD. But yeah, you'd still need one of the others to deal with external resources where the lifetime is not a specific stack frame. --=20 Marco
Nov 01 2014
parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Sat, 1 Nov 2014 19:10:15 +0100
Marco Leise via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 - reference counted object (doesn't exist in D)
but it exists! and with a size of machine pointer! i swear it exists, i'm using it!
Nov 01 2014
prev sibling parent "Kagamin" <spam here.lot> writes:
On Friday, 31 October 2014 at 04:28:47 UTC, Marco Leise wrote:
 How would one write a big desktop application when
 a simple animation can exhaust system memory in seconds ?
Yep, it can be a long way from PoC to a quality implementation. Write GtkD 2.0 with careful approach w.r.t. resource management. Though, reference counting may not be a silver bullet, it still requires you to not forget to clear unneeded resources. And it's of no use if the reference is in an GC-managed object. Also you can still create big resources too eagerly instead of on demand, and hold onto them for longer than needed.
Oct 31 2014