www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Re: Conspiracy Theory #1

reply Sean Kelly <sean invisibleduck.org> writes:
Travis Boucher Wrote:

 
 The fast, highly optimized web code is a very niche market.



I'm not sure it will remain this way for long.  Look at social networking
sites, where people spend a great deal of their time in what are essentially
user-created apps.  Make them half as efficient and the "cloud" will need twice
the resources to run them.
Nov 19 2009
next sibling parent reply Travis Boucher <boucher.travis gmail.com> writes:
Sean Kelly wrote:

 Travis Boucher Wrote:

 The fast, highly optimized web code is a very niche market.



 I'm not sure it will remain this way for long.  Look at social networking
sites, where people spend a great deal of their time in what are essentially
user-created apps.  Make them half as efficient and the "cloud" will need twice
the resources to run them.




I hope it doesn't remain this way.  Personally I am sick of fixing 
broken PHP code, retarded ruby code, and bad SQL queries.  However, the 
issue isn't the language as much as it is the coders.

Easy powerful languages = stupid coders who do stupid things.

D is an easy, powerful language, but has one aspect which may protect it 
against stupid coders.  Its hard to do stupid things in D.  Its harder 
to create a memory leak in D then it is to prevent one in C.

Hell, I've seen ruby do things which personally I thought was a memory 
leak at first, to later realize it was just a poor GC implementation. 
(this is mats ruby, not jruby or rubinius).

I know stupid coders will always exist, but D promotes good practice 
without sacrificing performance.
Nov 19 2009
next sibling parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Travis Boucher (boucher.travis gmail.com)'s article

 Sean Kelly wrote:
  Its harder
 to create a memory leak in D then it is to prevent one in C.



void doStuff() {
    uint[] foo = new uint[100_000_000];
}

void main() {
    while(true) {
        doStuff();
    }
}
Nov 19 2009
parent reply Travis Boucher <boucher.travis gmail.com> writes:
dsimcha wrote:

 == Quote from Travis Boucher (boucher.travis gmail.com)'s article

 Sean Kelly wrote:
  Its harder
 to create a memory leak in D then it is to prevent one in C.



 void doStuff() {
     uint[] foo = new uint[100_000_000];
 }
 
 void main() {
     while(true) {
         doStuff();
     }
 }
 



Hmm, that seems like that should be an implementation bug.  Shouldn't 
foo be marked for GC once it scope?  (I have never used new on a 
primitive type, so I don't know)
Nov 19 2009
parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Travis Boucher (boucher.travis gmail.com)'s article

 dsimcha wrote:

 == Quote from Travis Boucher (boucher.travis gmail.com)'s article

 Sean Kelly wrote:
  Its harder
 to create a memory leak in D then it is to prevent one in C.



 void doStuff() {
     uint[] foo = new uint[100_000_000];
 }

 void main() {
     while(true) {
         doStuff();
     }
 }



 foo be marked for GC once it scope?  (I have never used new on a
 primitive type, so I don't know)



It's conservative GC.  D's GC, along with the Hans Boehm GC and probably most
GCs
for close to the metal languages, can't perfectly identify what's a pointer and
what's not.  Therefore, for sufficiently large allocations there's a high
probability that some bit pattern that looks like a pointer but isn't one will
keep the allocation alive long after there are no "real" references to it left.
Nov 20 2009
parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Denis Koroskin (2korden gmail.com)'s article

 On Fri, 20 Nov 2009 17:28:07 +0300, dsimcha <dsimcha yahoo.com> wrote:

 == Quote from Travis Boucher (boucher.travis gmail.com)'s article

 dsimcha wrote:

 == Quote from Travis Boucher (boucher.travis gmail.com)'s article

 Sean Kelly wrote:
  Its harder
 to create a memory leak in D then it is to prevent one in C.



 void doStuff() {
     uint[] foo = new uint[100_000_000];
 }

 void main() {
     while(true) {
         doStuff();
     }
 }



 foo be marked for GC once it scope?  (I have never used new on a
 primitive type, so I don't know)



 It's conservative GC.  D's GC, along with the Hans Boehm GC and probably
 most GCs
 for close to the metal languages, can't perfectly identify what's a
 pointer and
 what's not.  Therefore, for sufficiently large allocations there's a high
 probability that some bit pattern that looks like a pointer but isn't
 one will
 keep the allocation alive long after there are no "real" references to
 it left.



 thought they aren't scanned for pointers (unlike, say, void[]).



Right, but they can still be the target of false pointers.  In this case, false
pointers keep each instance of foo[] alive, leading to severe memory leaks.
Nov 20 2009
next sibling parent reply Travis Boucher <boucher.travis gmail.com> writes:
dsimcha wrote:

 == Quote from Denis Koroskin (2korden gmail.com)'s article

 On Fri, 20 Nov 2009 17:28:07 +0300, dsimcha <dsimcha yahoo.com> wrote:

 == Quote from Travis Boucher (boucher.travis gmail.com)'s article

 dsimcha wrote:

 == Quote from Travis Boucher (boucher.travis gmail.com)'s article

 Sean Kelly wrote:
  Its harder
 to create a memory leak in D then it is to prevent one in C.



     uint[] foo = new uint[100_000_000];
 }

 void main() {
     while(true) {
         doStuff();
     }
 }



 foo be marked for GC once it scope?  (I have never used new on a
 primitive type, so I don't know)



 most GCs
 for close to the metal languages, can't perfectly identify what's a
 pointer and
 what's not.  Therefore, for sufficiently large allocations there's a high
 probability that some bit pattern that looks like a pointer but isn't
 one will
 keep the allocation alive long after there are no "real" references to
 it left.



 thought they aren't scanned for pointers (unlike, say, void[]).



 Right, but they can still be the target of false pointers.  In this case, false
 pointers keep each instance of foo[] alive, leading to severe memory leaks.



But the issue is more of a GC implementation issue then a language 
issue, correct?  Or is this an issue of all lower level language garbage 
collectors?

I do not know much about GC, just basic concepts.
Nov 20 2009
parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Travis Boucher (boucher.travis gmail.com)'s article

 dsimcha wrote:

 == Quote from Denis Koroskin (2korden gmail.com)'s article

 On Fri, 20 Nov 2009 17:28:07 +0300, dsimcha <dsimcha yahoo.com> wrote:

 == Quote from Travis Boucher (boucher.travis gmail.com)'s article

 dsimcha wrote:

 == Quote from Travis Boucher (boucher.travis gmail.com)'s article

 Sean Kelly wrote:
  Its harder
 to create a memory leak in D then it is to prevent one in C.



     uint[] foo = new uint[100_000_000];
 }

 void main() {
     while(true) {
         doStuff();
     }
 }



 foo be marked for GC once it scope?  (I have never used new on a
 primitive type, so I don't know)



 most GCs
 for close to the metal languages, can't perfectly identify what's a
 pointer and
 what's not.  Therefore, for sufficiently large allocations there's a high
 probability that some bit pattern that looks like a pointer but isn't
 one will
 keep the allocation alive long after there are no "real" references to
 it left.



 thought they aren't scanned for pointers (unlike, say, void[]).



 Right, but they can still be the target of false pointers.  In this case, false
 pointers keep each instance of foo[] alive, leading to severe memory leaks.



 issue, correct?



Yes.


 Or is this an issue of all lower level language garbage
 collectors?



Kinda sorta.  It's possible, but not easy, to implement fully precise GC (except
for the extreme corner case of unions of reference and non-reference types) in a
close to the metal, statically compiled language.
Nov 20 2009
next sibling parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Denis Koroskin (2korden gmail.com)'s article

 On Fri, 20 Nov 2009 19:24:05 +0300, dsimcha <dsimcha yahoo.com> wrote:

 == Quote from Travis Boucher (boucher.travis gmail.com)'s article

 dsimcha wrote:

 == Quote from Denis Koroskin (2korden gmail.com)'s article

 On Fri, 20 Nov 2009 17:28:07 +0300, dsimcha <dsimcha yahoo.com>







 == Quote from Travis Boucher (boucher.travis gmail.com)'s article

 dsimcha wrote:

 == Quote from Travis Boucher (boucher.travis gmail.com)'s article

 Sean Kelly wrote:
  Its harder
 to create a memory leak in D then it is to prevent one in C.



     uint[] foo = new uint[100_000_000];
 }

 void main() {
     while(true) {
         doStuff();
     }
 }
















 foo be marked for GC once it scope?  (I have never used new on a
 primitive type, so I don't know)













 most GCs
 for close to the metal languages, can't perfectly identify what's a
 pointer and
 what's not.  Therefore, for sufficiently large allocations there's










 probability that some bit pattern that looks like a pointer but










 one will
 keep the allocation alive long after there are no "real" references










 it left.



 thought they aren't scanned for pointers (unlike, say, void[]).



 Right, but they can still be the target of false pointers.  In this




 pointers keep each instance of foo[] alive, leading to severe memory



 But the issue is more of a GC implementation issue then a language
 issue, correct?



 Yes.


 Or is this an issue of all lower level language garbage
 collectors?



 Kinda sorta.  It's possible, but not easy, to implement fully precise GC
 (except
 for the extreme corner case of unions of reference and non-reference
 types) in a
 close to the metal, statically compiled language.



 Cyclone http://cyclone.thelanguage.org/wiki/Tagged%20Unions). Would that
 help?



It would be negligible.  The idea is that unions of reference and non-reference
types are such a corner case that they could be handled conservatively as a
special case, and then it's possible, at least in principle, to deal with the
other 99.99999% of cases precisely and being conservative in 0.00001% of cases
is
really of no practical significance.

Keep in mind that we would need to have the ability to pin and scan
conservatively
anyhow, since a systems language must allow allocation of untyped blocks of
memory.

I guess what I should have said is that the SafeD subset can be made 100%
precise
and D as a whole can be made about 99+% precise.
Nov 20 2009
parent reply Rainer Deyke <rainerd eldwood.com> writes:
dsimcha wrote:

 == Quote from Denis Koroskin (2korden gmail.com)'s article
 It would be negligible.  The idea is that unions of reference and non-reference
 types are such a corner case that they could be handled conservatively as a
 special case, and then it's possible, at least in principle, to deal with the
 other 99.99999% of cases precisely and being conservative in 0.00001% of cases
is
 really of no practical significance.



Yes, but a moving GC needs to be 100% precise, not 99.99999%.


-- 
Rainer Deyke - rainerd eldwood.com
Nov 20 2009
parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Rainer Deyke (rainerd eldwood.com)'s article

 dsimcha wrote:

 == Quote from Denis Koroskin (2korden gmail.com)'s article
 It would be negligible.  The idea is that unions of reference and non-reference
 types are such a corner case that they could be handled conservatively as a
 special case, and then it's possible, at least in principle, to deal with the
 other 99.99999% of cases precisely and being conservative in 0.00001% of cases
is
 really of no practical significance.






Not if you allow pinning, which we'd need anyhow for untyped, conservatively
scanned memory blocks.
Nov 20 2009
parent reply Rainer Deyke <rainerd eldwood.com> writes:
dsimcha wrote:

 == Quote from Rainer Deyke (rainerd eldwood.com)'s article

 Yes, but a moving GC needs to be 100% precise, not 99.99999%.



 Not if you allow pinning, which we'd need anyhow for untyped, conservatively
 scanned memory blocks.



If you allow pinning then you no longer get the full benefits of a
moving gc.  It would be nice to be able to trade untyped, conservatively
scanned memory blocks for a better gc.


-- 
Rainer Deyke - rainerd eldwood.com
Nov 20 2009
parent Lutger <lutger.blijdestijn gmail.com> writes:
Rainer Deyke wrote:


 dsimcha wrote:

 == Quote from Rainer Deyke (rainerd eldwood.com)'s article

 Yes, but a moving GC needs to be 100% precise, not 99.99999%.



 Not if you allow pinning, which we'd need anyhow for untyped,
 conservatively scanned memory blocks.



 If you allow pinning then you no longer get the full benefits of a
 moving gc.  It would be nice to be able to trade untyped, conservatively
 scanned memory blocks for a better gc.
 
 



Is it possible to allocate 'pinnable' objects from a different heap and 
still have your normal objects managed by an optimal moving gc?
Nov 21 2009
prev sibling next sibling parent Travis Boucher <boucher.travis gmail.com> writes:
Denis Koroskin wrote:

 On Fri, 20 Nov 2009 19:24:05 +0300, dsimcha <dsimcha yahoo.com> wrote:
 

 == Quote from Travis Boucher (boucher.travis gmail.com)'s article

 dsimcha wrote:

 == Quote from Denis Koroskin (2korden gmail.com)'s article

 On Fri, 20 Nov 2009 17:28:07 +0300, dsimcha <dsimcha yahoo.com> 







 == Quote from Travis Boucher (boucher.travis gmail.com)'s article

 dsimcha wrote:

 == Quote from Travis Boucher (boucher.travis gmail.com)'s article

 Sean Kelly wrote:
  Its harder
 to create a memory leak in D then it is to prevent one in C.



     uint[] foo = new uint[100_000_000];
 }

 void main() {
     while(true) {
         doStuff();
     }
 }
















 foo be marked for GC once it scope?  (I have never used new on a
 primitive type, so I don't know)













 most GCs
 for close to the metal languages, can't perfectly identify what's a
 pointer and
 what's not.  Therefore, for sufficiently large allocations 










 probability that some bit pattern that looks like a pointer but 










 one will
 keep the allocation alive long after there are no "real" 










 it left.



 thought they aren't scanned for pointers (unlike, say, void[]).



 Right, but they can still be the target of false pointers.  In this 




 pointers keep each instance of foo[] alive, leading to severe 



 But the issue is more of a GC implementation issue then a language
 issue, correct?



 Yes.


 Or is this an issue of all lower level language garbage
 collectors?



 Kinda sorta.  It's possible, but not easy, to implement fully precise 
 GC (except
 for the extreme corner case of unions of reference and non-reference 
 types) in a
 close to the metal, statically compiled language.



 Unions could be deprecated in favor of tagged unions (see an example in 
 Cyclone http://cyclone.thelanguage.org/wiki/Tagged%20Unions). Would that 
 help?



Probably not since the bit pattern of int i could still match a valid 
pointer.

Foo.i = cast(int)&Foo;   // for bad practice ugliness

or Foo.i = (some expression that happens to equal &Foo)


Adding extra information to a union could also have the bad side effect 
of killing performance as writes would include an extra write, and 
additional memory would be required (which would cause another set of 
issues on how to handle alignment).
Nov 20 2009
prev sibling parent Travis Boucher <boucher.travis gmail.com> writes:
Leandro Lucarella wrote:

 dsimcha, el 20 de noviembre a las 16:24 me escribiste:

 Right, but they can still be the target of false pointers.  In this case, false
 pointers keep each instance of foo[] alive, leading to severe memory leaks.



 issue, correct?





 Or is this an issue of all lower level language garbage
 collectors?



 for the extreme corner case of unions of reference and non-reference types) in
a
 close to the metal, statically compiled language.



 I don't think so if you want to be able to link to C code, unless I'm
 missing something...
 



The extern (C) stuff and malloc allocated memory isn't garbage collected.
Nov 20 2009
prev sibling parent reply "Nick Sabalausky" <a a.a> writes:
"dsimcha" <dsimcha yahoo.com> wrote in message 
news:he6aah$4d6$1 digitalmars.com...

 == Quote from Denis Koroskin (2korden gmail.com)'s article

 Aren't uint array allocations have hasPointers flag set off? I always
 thought they aren't scanned for pointers (unlike, say, void[]).



 Right, but they can still be the target of false pointers.  In this case, 
 false
 pointers keep each instance of foo[] alive, leading to severe memory 
 leaks.



I don't suppose there's a way to lookup the pointers the GC believes it has 
found to a given piece of GC-ed memory? Sounds like that would be very 
useful, if not essential, for debugging/optimizing memory usage.
Nov 21 2009
parent Travis Boucher <boucher.travis gmail.com> writes:
Nick Sabalausky wrote:

 "dsimcha" <dsimcha yahoo.com> wrote in message 
 news:he6aah$4d6$1 digitalmars.com...

 == Quote from Denis Koroskin (2korden gmail.com)'s article

 Aren't uint array allocations have hasPointers flag set off? I always
 thought they aren't scanned for pointers (unlike, say, void[]).



 false
 pointers keep each instance of foo[] alive, leading to severe memory 
 leaks.



 I don't suppose there's a way to lookup the pointers the GC believes it has 
 found to a given piece of GC-ed memory? Sounds like that would be very 
 useful, if not essential, for debugging/optimizing memory usage.
 
 



Maybe extend the GC interface so the compiler and language in general 
will give hints on what the memory is being used for.  This could even 
be extended to application code as well.

MEM_OBJECT, MEM_STRUCT, MEM_PTRARRY, MEM_ARRAY, etc. (I haven't fully 
thought this through so these examples may be bad).

Then the GC implementations can decide how to allocate the memory in the 
  best way for the underlying architecture.  I know this would be useful 
on weird memory layouts found in embedded machines (NDS for example), 
but could also be extended language-wise to other hardware memory areas. 
  For example, allocating memory on video cards or DSP hardware.

Like I said, this isn't something I have thought through much, and I 
don't know how much (if any) compiler/GC interface support would be 
required.
Nov 21 2009
prev sibling next sibling parent "Denis Koroskin" <2korden gmail.com> writes:
On Fri, 20 Nov 2009 17:28:07 +0300, dsimcha <dsimcha yahoo.com> wrote:


 == Quote from Travis Boucher (boucher.travis gmail.com)'s article

 dsimcha wrote:

 == Quote from Travis Boucher (boucher.travis gmail.com)'s article

 Sean Kelly wrote:
  Its harder
 to create a memory leak in D then it is to prevent one in C.



 void doStuff() {
     uint[] foo = new uint[100_000_000];
 }

 void main() {
     while(true) {
         doStuff();
     }
 }



 foo be marked for GC once it scope?  (I have never used new on a
 primitive type, so I don't know)



 It's conservative GC.  D's GC, along with the Hans Boehm GC and probably  
 most GCs
 for close to the metal languages, can't perfectly identify what's a  
 pointer and
 what's not.  Therefore, for sufficiently large allocations there's a high
 probability that some bit pattern that looks like a pointer but isn't  
 one will
 keep the allocation alive long after there are no "real" references to  
 it left.



Aren't uint array allocations have hasPointers flag set off? I always  
thought they aren't scanned for pointers (unlike, say, void[]).
Nov 20 2009
prev sibling next sibling parent "Denis Koroskin" <2korden gmail.com> writes:
On Fri, 20 Nov 2009 19:24:05 +0300, dsimcha <dsimcha yahoo.com> wrote:


 == Quote from Travis Boucher (boucher.travis gmail.com)'s article

 dsimcha wrote:

 == Quote from Denis Koroskin (2korden gmail.com)'s article

 On Fri, 20 Nov 2009 17:28:07 +0300, dsimcha <dsimcha yahoo.com>  







 == Quote from Travis Boucher (boucher.travis gmail.com)'s article

 dsimcha wrote:

 == Quote from Travis Boucher (boucher.travis gmail.com)'s article

 Sean Kelly wrote:
  Its harder
 to create a memory leak in D then it is to prevent one in C.



     uint[] foo = new uint[100_000_000];
 }

 void main() {
     while(true) {
         doStuff();
     }
 }
















 foo be marked for GC once it scope?  (I have never used new on a
 primitive type, so I don't know)













 most GCs
 for close to the metal languages, can't perfectly identify what's a
 pointer and
 what's not.  Therefore, for sufficiently large allocations there's  










 probability that some bit pattern that looks like a pointer but  










 one will
 keep the allocation alive long after there are no "real" references  










 it left.



 thought they aren't scanned for pointers (unlike, say, void[]).



 Right, but they can still be the target of false pointers.  In this  




 pointers keep each instance of foo[] alive, leading to severe memory  



 But the issue is more of a GC implementation issue then a language
 issue, correct?



 Yes.


 Or is this an issue of all lower level language garbage
 collectors?



 Kinda sorta.  It's possible, but not easy, to implement fully precise GC  
 (except
 for the extreme corner case of unions of reference and non-reference  
 types) in a
 close to the metal, statically compiled language.



Unions could be deprecated in favor of tagged unions (see an example in  
Cyclone http://cyclone.thelanguage.org/wiki/Tagged%20Unions). Would that  
help?
Nov 20 2009
prev sibling next sibling parent Leandro Lucarella <llucax gmail.com> writes:
Travis Boucher, el 20 de noviembre a las 16:45 me escribiste:

 Leandro Lucarella wrote:

dsimcha, el 20 de noviembre a las 16:24 me escribiste:

Right, but they can still be the target of false pointers.  In this case, false
pointers keep each instance of foo[] alive, leading to severe memory leaks.



issue, correct?





Or is this an issue of all lower level language garbage
collectors?



for the extreme corner case of unions of reference and non-reference types) in a
close to the metal, statically compiled language.



I don't think so if you want to be able to link to C code, unless I'm
missing something...



 The extern (C) stuff and malloc allocated memory isn't garbage collected.



I know, but the stack is used as a root of the live data, and if you use
C code, you will have frames without type information. So you will never
get full a full precise root set, unless you find some way to separate the
D stack from the C stack and completely ignore the C stack. Again, unless
I'm missing something :)

-- 
Leandro Lucarella (AKA luca)                     http://llucax.com.ar/
----------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145  104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------
- Mire, don Inodoro! Una paloma con un anillo en la pata! Debe ser
  mensajera y cayó aquí!
- Y... si no es mensajera es coqueta... o casada.
	-- Mendieta e Inodoro Pereyra
Nov 20 2009
prev sibling parent "Denis Koroskin" <2korden gmail.com> writes:
On Sat, 21 Nov 2009 05:57:48 +0300, Rainer Deyke <rainerd eldwood.com>  
wrote:


 dsimcha wrote:

 == Quote from Rainer Deyke (rainerd eldwood.com)'s article

 Yes, but a moving GC needs to be 100% precise, not 99.99999%.



 Not if you allow pinning, which we'd need anyhow for untyped,  
 conservatively
 scanned memory blocks.



 If you allow pinning then you no longer get the full benefits of a
 moving gc.  It would be nice to be able to trade untyped, conservatively
 scanned memory blocks for a better gc.



Pinning is a must-have if you want communicate with code written in other  
languages (C, for example).
Casting from Object to void* would be forbidden, use void* pinnedAddress =  
GC.pin(obj); (or similar) instead.
Nov 21 2009
prev sibling parent Leandro Lucarella <llucax gmail.com> writes:
dsimcha, el 20 de noviembre a las 16:24 me escribiste:

 Right, but they can still be the target of false pointers.  In this case, false
 pointers keep each instance of foo[] alive, leading to severe memory leaks.



 issue, correct?



 Yes.
 

 Or is this an issue of all lower level language garbage
 collectors?



 Kinda sorta.  It's possible, but not easy, to implement fully precise GC
(except
 for the extreme corner case of unions of reference and non-reference types) in
a
 close to the metal, statically compiled language.



I don't think so if you want to be able to link to C code, unless I'm
missing something...

-- 
Leandro Lucarella (AKA luca)                     http://llucax.com.ar/
----------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145  104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------
"Lidiar" no es lo mismo que "holguear"; ya que "lidiar" es relativo a
"lidia" y "holguear" es relativo a "olga".
	-- Ricardo Vaporeso
Nov 20 2009