www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Knowledge of managed memory pointers

reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
--089e015370883c973004f73784d5
Content-Type: text/plain; charset=UTF-8

It occurs to me that a central issue regarding the memory management
debate, and a major limiting factor with respect to options, is the fact
that, currently, it's impossible to tell a raw pointer apart from a gc
pointer.

Is this is a problem worth solving? And would it be as big an enabler to
address some tricky problems as it seems to be at face value?

What are some options? Without turning to fat pointers or convoluted
changes in the type system, are there any clever mechanisms that could be
applied to distinguish managed from unmanaged pointers. If an API could be
provided in druntime, it may be used by GC's, ARC, allocators, or systems
that operate at the barrier between languages.

Obviously it needs to be super trivial to gather this information from the
pointer...
On embedded systems with fixed/limited memory it's easy, just make the gc
allocate pages in course physically aligned blocks and check a bit indexed
by a couple of bits in pointers whether that page is owned by the GC or not.

On large scale OS's with unknown quantity (perhaps lots) of memory, it's
not so simple, but they have other advantages, like virtual memory
managers. Can virtual pages be attributed with a bit of data somehow that's
easy to look up?

What about 'hacks' like an unlikely sentinel value at ptr[-1]?

--089e015370883c973004f73784d5
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">It occurs to me that a central issue regarding the memory =
management debate, and a major limiting factor with respect to options, is =
the fact that, currently, it&#39;s impossible to tell a raw pointer apart f=
rom a gc pointer.<div>
<br></div><div>Is this is a problem worth solving? And would it be as big a=
n enabler to address some tricky problems as it seems to be at face value?<=
/div><div><br></div><div>What are some options? Without turning to fat poin=
ters or convoluted changes in the type system, are there any clever mechani=
sms that could be applied to distinguish managed from unmanaged pointers. I=
f an API could be provided in druntime, it may be used by GC&#39;s, ARC, al=
locators, or systems that operate at the barrier between languages.</div>
<div><br></div><div>Obviously it needs to be super trivial to gather this i=
nformation from the pointer...</div><div>On embedded systems with fixed/lim=
ited memory it&#39;s easy, just make the gc allocate pages in course physic=
ally aligned blocks and check a bit indexed by a couple of bits in pointers=
 whether that page is owned by the GC or not.</div>
<div><br></div><div>On large scale OS&#39;s with unknown quantity (perhaps =
lots) of memory, it&#39;s not so simple, but they have other advantages, li=
ke virtual memory managers. Can virtual pages be attributed with a bit of d=
ata somehow that&#39;s easy to look up?</div>
<div><br></div><div>What about &#39;hacks&#39; like an unlikely sentinel va=
lue at ptr[-1]?</div></div>

--089e015370883c973004f73784d5--
Apr 16 2014
next sibling parent "Kagamin" <spam here.lot> writes:
You can do anything, what fits your task, see RefCounted and 
Unique for an example on how to write smart pointers.
Apr 17 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
--001a11333b2034b64604f73c5481
Content-Type: text/plain; charset=UTF-8

On 17 April 2014 18:20, Kagamin via Digitalmars-d <
digitalmars-d puremagic.com> wrote:

 You can do anything, what fits your task, see RefCounted and Unique for an
 example on how to write smart pointers.

... what? I don't think you understood my post. void f(void* ptr) { // was ptr allocated with malloc, or new? } If we knew this, we may be able to address some problems with designing better GC's, or cross-language API's. --001a11333b2034b64604f73c5481 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On 1= 7 April 2014 18:20, Kagamin via Digitalmars-d <span dir=3D"ltr">&lt;<a href= =3D"mailto:digitalmars-d puremagic.com" target=3D"_blank">digitalmars-d pur= emagic.com</a>&gt;</span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex">You can do anything, what fits your task, se= e RefCounted and Unique for an example on how to write smart pointers.<br> </blockquote></div><br></div><div class=3D"gmail_extra">... what?</div><div= class=3D"gmail_extra"><br></div><div class=3D"gmail_extra">I don&#39;t thi= nk you understood my post.</div><div class=3D"gmail_extra"><br></div><div c= lass=3D"gmail_extra"> void f(void* ptr)</div><div class=3D"gmail_extra">{</div><div class=3D"gmai= l_extra">=C2=A0 // was ptr allocated with malloc, or new?</div><div class= =3D"gmail_extra">}</div><div class=3D"gmail_extra"><br></div><div class=3D"= gmail_extra"> If we knew this, we may be able to address some problems with designing bet= ter GC&#39;s, or cross-language API&#39;s.</div></div> --001a11333b2034b64604f73c5481--
Apr 17 2014
prev sibling next sibling parent Orvid King via Digitalmars-d <digitalmars-d puremagic.com> writes:
I think the biggest advantage to this distinction would really be the
cross-language API's, the GC can determine which pointers it owns,
although I don't believe it currently exposes this capability.

On 4/17/14, Manu via Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On 17 April 2014 18:20, Kagamin via Digitalmars-d <
 digitalmars-d puremagic.com> wrote:

 You can do anything, what fits your task, see RefCounted and Unique for
 an
 example on how to write smart pointers.

... what? I don't think you understood my post. void f(void* ptr) { // was ptr allocated with malloc, or new? } If we knew this, we may be able to address some problems with designing better GC's, or cross-language API's.

Apr 17 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
--089e011828e436f13804f73d5343
Content-Type: text/plain; charset=UTF-8

On 17 April 2014 23:14, Orvid King via Digitalmars-d <
digitalmars-d puremagic.com> wrote:

 I think the biggest advantage to this distinction would really be the
 cross-language API's, the GC can determine which pointers it owns,
 although I don't believe it currently exposes this capability.

But in a lightning fast way? Let's imagine ARC refcounts were stored at ptr[-1], how can we know if this is a managed pointer or not? I think the major hurdle in an ARC implementation is distinguishing a managed pointer from a malloc pointer without making (breaking?) changes to the type system. I would also find this useful in language boundary interaction though. I have had numerous issues identifying proper handling of cross-language memory. --089e011828e436f13804f73d5343 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On 1= 7 April 2014 23:14, Orvid King via Digitalmars-d <span dir=3D"ltr">&lt;<a h= ref=3D"mailto:digitalmars-d puremagic.com" target=3D"_blank">digitalmars-d = puremagic.com</a>&gt;</span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex">I think the biggest advantage to this distin= ction would really be the<br> cross-language API&#39;s, the GC can determine which pointers it owns,<br> although I don&#39;t believe it currently exposes this capability.<br></blo= ckquote><div><br></div><div>But in a lightning fast way? Let&#39;s imagine = ARC refcounts were stored at ptr[-1], how can we know if this is a managed = pointer or not?</div> <div>I think the major hurdle in an ARC implementation is distinguishing a = managed pointer from a malloc pointer without making (breaking?) changes to= the type system.</div><div><br></div><div>I would also find this useful in= language boundary interaction though. I have had numerous issues identifyi= ng proper handling of cross-language memory.</div> </div></div></div> --089e011828e436f13804f73d5343--
Apr 17 2014
prev sibling next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 04/17/2014 08:55 AM, Manu via Digitalmars-d wrote:
 It occurs to me that a central issue regarding the memory management
 debate, and a major limiting factor with respect to options, is the fact
 that, currently, it's impossible to tell a raw pointer apart from a gc
 pointer.

 Is this is a problem worth solving? And would it be as big an enabler to
 address some tricky problems as it seems to be at face value?

 What are some options? Without turning to fat pointers or convoluted
 changes in the type system, are there any clever mechanisms that could
 be applied to distinguish managed from unmanaged pointers.

It does not matter if changes to the type system are 'convoluted'. (They don't need to be.)
 If an API
 could be provided in druntime, it may be used by GC's, ARC, allocators,
 or systems that operate at the barrier between languages.

There already is. bool isGCPointer(void* ptr){ import core.memory; return !!GC.addrOf(ptr); } void main(){ import std.c.stdlib; auto x=cast(int*)malloc(int.sizeof); auto y=new int; assert(!x.isGCPointer() && y.isGCPointer()); }
Apr 17 2014
parent Jacob Carlborg <doob me.com> writes:
On 2014-04-17 16:59, Steven Schveighoffer wrote:

 I don't think this is a viable mechanism to check pointers. It's too slow.

Couldn't a single bit be used to indicate if it's a GC pointer or not? -- /Jacob Carlborg
Apr 19 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 17 Apr 2014 10:52:19 -0400, Timon Gehr <timon.gehr gmx.ch> wrote:

 On 04/17/2014 08:55 AM, Manu via Digitalmars-d wrote:

 If an API
 could be provided in druntime, it may be used by GC's, ARC, allocators,
 or systems that operate at the barrier between languages.

There already is. bool isGCPointer(void* ptr){ import core.memory; return !!GC.addrOf(ptr); }

I don't think this is a viable mechanism to check pointers. It's too slow. -Steve
Apr 17 2014
prev sibling next sibling parent "Kagamin" <spam here.lot> writes:
On Thursday, 17 April 2014 at 12:39:59 UTC, Manu via 
Digitalmars-d wrote:
 void f(void* ptr)
 {
   // was ptr allocated with malloc, or new?

Then what?
Apr 17 2014
prev sibling next sibling parent "Kagamin" <spam here.lot> writes:
On Thursday, 17 April 2014 at 14:59:14 UTC, Steven Schveighoffer 
wrote:
 I don't think this is a viable mechanism to check pointers. 
 It's too slow.

I suggested to write a smart pointer. It could provide compile-time checks and whatever developer feels like.
Apr 17 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
--047d7b5d302a802a1904f745d88b
Content-Type: text/plain; charset=UTF-8

On 18 April 2014 04:10, Kagamin via Digitalmars-d <
digitalmars-d puremagic.com> wrote:

 On Thursday, 17 April 2014 at 12:39:59 UTC, Manu via Digitalmars-d wrote:

 void f(void* ptr)
 {
   // was ptr allocated with malloc, or new?

Then what?

Whatever. inc/dec ref, or not. core.memory.addRoot/Range, or not. That sort of thing. There's a persistent problem when dealing with memory systems in D that it must remain backward compatible with C and raw pointers, and that creates complications. --047d7b5d302a802a1904f745d88b Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On 1= 8 April 2014 04:10, Kagamin via Digitalmars-d <span dir=3D"ltr">&lt;<a href= =3D"mailto:digitalmars-d puremagic.com" target=3D"_blank">digitalmars-d pur= emagic.com</a>&gt;</span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"><div class=3D"">On Thursday, 17 April 2014 a= t 12:39:59 UTC, Manu via Digitalmars-d wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> void f(void* ptr)<br> {<br> =C2=A0 // was ptr allocated with malloc, or new?<br> </blockquote> <br></div> Then what?<br> </blockquote></div><br></div><div class=3D"gmail_extra">Whatever. inc/dec r= ef, or not. core.memory.addRoot/Range, or not. That sort of thing.</div><di= v class=3D"gmail_extra"><br></div><div class=3D"gmail_extra">There&#39;s a = persistent problem =C2=A0when dealing with memory systems in D that it must= remain backward compatible with C and raw pointers, and that creates compl= ications.</div> </div> --047d7b5d302a802a1904f745d88b--
Apr 17 2014
prev sibling next sibling parent "Tove" <tove fransson.se> writes:
On Friday, 18 April 2014 at 00:01:25 UTC, Manu via Digitalmars-d 
wrote:
 On 18 April 2014 04:10, Kagamin via Digitalmars-d <
 digitalmars-d puremagic.com> wrote:

 On Thursday, 17 April 2014 at 12:39:59 UTC, Manu via 
 Digitalmars-d wrote:

 void f(void* ptr)
 {
   // was ptr allocated with malloc, or new?

Then what?

Whatever. inc/dec ref, or not. core.memory.addRoot/Range, or not. That sort of thing. There's a persistent problem when dealing with memory systems in D that it must remain backward compatible with C and raw pointers, and that creates complications.

Both NaCl and go-lang uses a trick to reserve a huge amount of continuos virtual memory...
Apr 18 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
--001a11346e9a34179904f74e7cb7
Content-Type: text/plain; charset=UTF-8

On 18 April 2014 20:10, Tove via Digitalmars-d
<digitalmars-d puremagic.com>wrote:

 On Friday, 18 April 2014 at 00:01:25 UTC, Manu via Digitalmars-d wrote:

 On 18 April 2014 04:10, Kagamin via Digitalmars-d <
 digitalmars-d puremagic.com> wrote:

  On Thursday, 17 April 2014 at 12:39:59 UTC, Manu via Digitalmars-d wrote:
  void f(void* ptr)
 {
   // was ptr allocated with malloc, or new?


sort of thing. There's a persistent problem when dealing with memory systems in D that it must remain backward compatible with C and raw pointers, and that creates complications.

Both NaCl and go-lang uses a trick to reserve a huge amount of continuos virtual memory...

Yeah, that's what I imagined too... but does that work in 32bit apps? --001a11346e9a34179904f74e7cb7 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On 1= 8 April 2014 20:10, Tove via Digitalmars-d <span dir=3D"ltr">&lt;<a href=3D= "mailto:digitalmars-d puremagic.com" target=3D"_blank">digitalmars-d purema= gic.com</a>&gt;</span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"><div class=3D"">On Friday, 18 April 2014 at = 00:01:25 UTC, Manu via Digitalmars-d wrote:<br> </div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-l= eft:1px #ccc solid;padding-left:1ex"><div class=3D""> On 18 April 2014 04:10, Kagamin via Digitalmars-d &lt;<br> <a href=3D"mailto:digitalmars-d puremagic.com" target=3D"_blank">digitalmar= s-d puremagic.com</a>&gt; wrote:<br> <br> </div><div><div class=3D"h5"><blockquote class=3D"gmail_quote" style=3D"mar= gin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> On Thursday, 17 April 2014 at 12:39:59 UTC, Manu via Digitalmars-d wrote:<b= r> <br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> void f(void* ptr)<br> {<br> =C2=A0 // was ptr allocated with malloc, or new?<br> <br> </blockquote> <br> Then what?<br> <br> </blockquote> <br></div></div><div class=3D""> Whatever. inc/dec ref, or not. core.memory.addRoot/Range, or not. That sort= <br> of thing.<br> <br> There&#39;s a persistent problem =C2=A0when dealing with memory systems in = D that it<br> must remain backward compatible with C and raw pointers, and that creates<b= r> complications.<br> </div></blockquote> <br> Both NaCl and go-lang uses a trick to reserve a huge amount of continuos vi= rtual memory...<br> </blockquote></div><br></div><div class=3D"gmail_extra">Yeah, that&#39;s wh= at I imagined too... but does that work in 32bit apps?</div></div> --001a11346e9a34179904f74e7cb7--
Apr 18 2014
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/16/2014 11:55 PM, Manu via Digitalmars-d wrote:
 It occurs to me that a central issue regarding the memory management debate,
and
 a major limiting factor with respect to options, is the fact that, currently,
 it's impossible to tell a raw pointer apart from a gc pointer.

No it isn't, in fact, the gc has to be able to tell the difference or it cannot do mark/sweep. It tells the difference with "does the pointer point into the gc memory pool."
 What about 'hacks' like an unlikely sentinel value at ptr[-1]?

That doesn't work as soon as one increments a pointer.
Apr 18 2014
prev sibling parent "Yota" <yotaxp thatGoogleMailThing.com> writes:
On Saturday, 19 April 2014 at 10:30:42 UTC, Jacob Carlborg wrote:
 On 2014-04-17 16:59, Steven Schveighoffer wrote:

 I don't think this is a viable mechanism to check pointers. 
 It's too slow.

Couldn't a single bit be used to indicate if it's a GC pointer or not?

I think he's hoping more for the static type system to provide the information. I think it would be nifty if the fancy layered allocators that were being designed would return pointers with types describing the allocator itself. Could this sort of info be useful in shared libs?
Apr 19 2014