www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - how do you mix ref counted types and non- nogc algorithms

reply ikod <geller.garry gmail.com> writes:
Hello,

I have ref-counted type (keeps reference to mallocated memory) 
which I would like to release as soon as possible (when I have no 
refs from my code).

The problem is - standard algorithms can use some GC allocations 
to keep refs to my data until GC is collected, so that I have no 
control over memory consumption when non- nogc algorithms applied 
to RC data.

Right now I have to avoid something like my_data.splitOn(), but 
is there any better solution?

Thanks
Jul 08 2020
next sibling parent user1234 <user1234 12.de> writes:
On Wednesday, 8 July 2020 at 11:26:20 UTC, ikod wrote:

 Hello,

 I have ref-counted type (keeps reference to mallocated memory) 
 which I would like to release as soon as possible (when I have 
 no refs from my code).

 The problem is - standard algorithms can use some GC 
 allocations to keep refs to my data until GC is collected, so 
 that I have no control over memory consumption when non- nogc


memory consumption is not your first problem (explained later)


 algorithms applied to RC data.

 Right now I have to avoid something like my_data.splitOn(), but 
 is there any better solution?

 Thanks


you must work with also on a `.dup`ed version, to leave your 
custom RC space, and later copy back the results in your ref 
counted space. Mixing two systems of memory is no joke. This can 
lead to crashes it at some point your own RC stuff are seen by 
the GC but not owned by a GC root.
Jul 08 2020
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 7/8/20 7:26 AM, ikod wrote:

 Hello,
 
 I have ref-counted type (keeps reference to mallocated memory) which I 
 would like to release as soon as possible (when I have no refs from my 
 code).
 
 The problem is - standard algorithms can use some GC allocations to keep 
 refs to my data until GC is collected, so that I have no control over 
 memory consumption when non- nogc algorithms applied to RC data.
 
 Right now I have to avoid something like my_data.splitOn(), but is there 
 any better solution?


My solution to this was to use the GC for ref counted data.

See here: http://schveiguy.github.io/iopipe/iopipe/refc/RefCounted.html

This way, the resources represented by the type (generally things like 
files or malloc'd items) are cleaned up, but the memory is still safe to 
use.

-Steve
Jul 08 2020
parent reply Avrina <avrina12309412342 gmail.com> writes:
On Wednesday, 8 July 2020 at 12:30:09 UTC, Steven Schveighoffer 
wrote:

 On 7/8/20 7:26 AM, ikod wrote:

 Hello,
 
 I have ref-counted type (keeps reference to mallocated memory) 
 which I would like to release as soon as possible (when I have 
 no refs from my code).
 
 The problem is - standard algorithms can use some GC 
 allocations to keep refs to my data until GC is collected, so 
 that I have no control over memory consumption when non- nogc 
 algorithms applied to RC data.
 
 Right now I have to avoid something like my_data.splitOn(), 
 but is there any better solution?


 My solution to this was to use the GC for ref counted data.

 See here: 
 http://schveiguy.github.io/iopipe/iopipe/refc/RefCounted.html

 This way, the resources represented by the type (generally 
 things like files or malloc'd items) are cleaned up, but the 
 memory is still safe to use.

 -Steve


My understanding to what he means is that a copy of RefCounted 
will be kept in GC'd memory. So the counter won't hit 0 until the 
GC collects and calls the destructor for that memory, which isn't 
guarantee'd to ever happen. Your implementation would have the 
same problem.
Jul 08 2020
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 7/8/20 11:27 AM, Avrina wrote:

 On Wednesday, 8 July 2020 at 12:30:09 UTC, Steven Schveighoffer wrote:

 On 7/8/20 7:26 AM, ikod wrote:

 Hello,

 I have ref-counted type (keeps reference to mallocated memory) which 
 I would like to release as soon as possible (when I have no refs from 
 my code).

 The problem is - standard algorithms can use some GC allocations to 
 keep refs to my data until GC is collected, so that I have no control 
 over memory consumption when non- nogc algorithms applied to RC data.

 Right now I have to avoid something like my_data.splitOn(), but is 
 there any better solution?


 My solution to this was to use the GC for ref counted data.

 See here: http://schveiguy.github.io/iopipe/iopipe/refc/RefCounted.html

 This way, the resources represented by the type (generally things like 
 files or malloc'd items) are cleaned up, but the memory is still safe 
 to use.


 
 My understanding to what he means is that a copy of RefCounted will be 
 kept in GC'd memory. So the counter won't hit 0 until the GC collects 
 and calls the destructor for that memory, which isn't guarantee'd to 
 ever happen. Your implementation would have the same problem.


Oh yeah, I misunderstood the problem.

That's going to be the case no matter what. The only true solution would 
be to use algorithms that use RefCounted instead of the GC. Which may be 
trading one problem for another.

-Steve
Jul 08 2020
parent ikod <geller.garry gmail.com> writes:
On Wednesday, 8 July 2020 at 16:28:02 UTC, Steven Schveighoffer 
wrote:

 On 7/8/20 11:27 AM, Avrina wrote:

 On Wednesday, 8 July 2020 at 12:30:09 UTC, Steven 
 Schveighoffer wrote:

 On 7/8/20 7:26 AM, ikod wrote:

 Hello,









 My understanding to what he means is that a copy of RefCounted 
 will be kept in GC'd memory. So the counter won't hit 0 until 
 the GC collects and calls the destructor for that memory, 
 which isn't guarantee'd to ever happen. Your implementation 
 would have the same problem.




Yes, this is the problem, thanks Avrina!


 Oh yeah, I misunderstood the problem.

 That's going to be the case no matter what. The only true 
 solution would be to use algorithms that use RefCounted instead 
 of the GC. Which may be trading one problem for another.


I see... Probably this can be configurable option (use GC or RC 
for algorithms internals) or can be inferred from the data type 
properties (like use RC if data methods used for algorithm are 
 nogc). But I'm not sure if this is possible and whether it makes 
practical sense.


 -Steve
Jul 08 2020
prev sibling parent Kagamin <spam here.lot> writes:
On Wednesday, 8 July 2020 at 11:26:20 UTC, ikod wrote:

 Right now I have to avoid something like my_data.splitOn(), but 
 is there any better solution?


You can lend a weak reference to GC structure, its lifetime will 
be shorter than that of remaining strong references.
Jul 10 2020