www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Re: Template Metaprogramming Made Easy (Huh?)

reply Justin Johansson <procode adam-dott-com.au> writes:
Nick Sabalausky Wrote:
 As someone who's been meaning to take a look at Scala, I'm very curious: 
 What did you dislike about it?

I'm somewhat reluctant to discuss Scala too much here as this is a D forum, but since more than one person asked it's only fair to reply with something. Now this is just my opinion, subjective and ignorant as it may be so, please, its horses for courses and YMMV. Also I fully support bio-diversity in programming languages and it's ridiculous to suggest that there is or should be only only true language and then to can a language because its not your idea of the one true language. Regarding my writing style, I attempt to employ a little humour (lame as it may be) so please take that into account before anyone decides to get really angry at me. Well here goes ... 1. Moved in with her for six months. She's an overkill of a language, seductive at first but then like a Black Widow spider eats her mate after courtship. http://en.wikipedia.org/wiki/Latrodectus_hesperus 2. Scala cannot make up her mind if she's a scripting language or serious language. Optional semicolons at the end of statements are really frustrating for code style consistency. Worst when sometimes you need them and you left them out everywhere else for code style consistency. (JavaScript has this problem too and JS guru Douglas Crockford recommends semicolons always be used in that language despite them being optional when statements clearly separated by newlines.) 3. Half-baked embeddable XML support in the language looks like she borrowed from ECMAScript's E4X. 4. Too many different ways of doing things. All very interesting and no doubt very clever but she needs to shave her hairy legs with Occam's razor before she starts to look like the sister of Frankenstein's monster**. http://en.wikipedia.org/wiki/Occam%27s_razor 4. Way too much of an academic approach to language design; appears to be a grand experiment to see how many academic papers can be derived from it. Google for "Scala experiment" with and without the quotes and you'll soon realize she was designed in a lab from a kitchen sink full of PL body parts like her brother in fiction**. Read Cedric's blog June 2008 for example http://beust.com/weblog/archives/000490.html Still there's no such thing as a "failed experiment". An experiment is just an "experiment". The light bulb wouldn't exist today if Edison wasn't as tenacious as he was. He simply treated "failed" experiments as just another way of learning how not to make a light bulb. 5. Newcomers to the language will find it's type system concepts overwhelming - co-variance and contra-variance etc. (don't know how D2 will address this better though). Yes these issues are important for OO libraries but feel there must be a more practical way out of the language complexity. Personally I always kept away from the hairy and scary bits of C++; you don't need 'em in a practical language. I've heard Scala's argument that all the complexity is hidden in the libraries so need to worry about it. Unfortunately I don't believe her. I learn a lot about a language by studying the library code and expect it to be as easy to read and understand as mainline code. 5. Not her fault (i.e. of the language), but after six months of courting Scala with the Eclipse plugin, suffering IDE crash after crash and lost code I just could not bring myself to suffering her any longer. Read August 2009 comments by Tim at http://blog.jayway.com/2009/03/12/scala-ide-support/ "Half a year after the above post, Iím still shocked at how badly the Scala plug-in for Eclipse behaves. Iíve downloaded several variants on Eclipse over the last half-year (currently 3.4.2), and NONE of them have been able to do even basic things reliably with the Scala plug-in (or vise-versa) ... without, say, basic knowledge of code (how to find a referenced method) and the ability to run more than five minutes, I might as well go back to Vim. (Which I think Iím going to have to do.)" When I went looking for an Eclipse plugin for D a few weeks ago, I soon discovered Descent. It's not perfect by any means, but works well enough not to be a thorn in your side and deserves credit for where it's at already. 6. In agreement with comments by Tony Arcieri (April 2009) "I was initially excited about Scala but slowly grew discontented with it." http://www.unlimitednovelty.com/2009/04/why-i-dont-like-scala.html To sum up, after six months of living in, I felt I still wasn't getting anywhere close to being intimate with her. It really shouldn't take that long to get up to speed with a new PL. My advice for D2 language designers is not to copy every "cool" feature of Scala. For good balance of ideas, look at Rich Hickey's Clojure language. http://clojure.org/
Sep 10 2009
next sibling parent Jeremie Pelletier <jeremiep gmail.com> writes:
Justin Johansson Wrote:
 5. Newcomers to the language will find it's type system concepts overwhelming
- co-variance and contra-variance etc.  (don't know how D2 will address this
better though). Yes these issues are important for OO libraries but feel there
must be a more practical way out of the language complexity.  Personally I
always kept away from the hairy and scary bits of C++; you don't need 'em in a
practical language.
 
 I've heard Scala's argument that all the complexity is hidden in the libraries
so need to worry about it.  Unfortunately I don't believe her.  I learn a lot
about a language by studying the library code and expect it to be as easy to
read and understand as mainline code.

I couldn't agree more, I learned how to use D by studying its runtime library over the past few years. To me it is especially useful to study a runtime library when it is used to implement features of the language, so you get a clear grasp of what using these features imply. I lost count on how many neat tricks I learned reading Andrei's metaprogramming code.
Sep 10 2009
prev sibling next sibling parent reply language_fan <foo bar.com.invalid> writes:
Thu, 10 Sep 2009 20:25:04 -0400, Justin Johansson thusly wrote:

 2. Scala cannot make up her mind if she's a scripting language or
 serious language.  Optional semicolons at the end of statements are
 really frustrating for code style consistency.  Worst when sometimes you
 need them and you left them out everywhere else for code style
 consistency.  (JavaScript has this problem too and JS guru Douglas
 Crockford recommends semicolons always be used in that language despite
 them being optional when statements clearly separated by newlines.)

Ok, so you did not like optional semicolons.
 
 3. Half-baked embeddable XML support in the language looks like she
 borrowed from ECMAScript's E4X.

Ok, borrowing features is bad.
 
 4. Too many different ways of doing things.  All very interesting and no
 doubt very clever but she needs to shave her hairy legs with Occam's
 razor before she starts to look like the sister of Frankenstein's
 monster**.

Examples? I can tell you that D has too many looping constructs: goto, for, foreach, foreach_reverse, do-while, while, recursive functions, recursive templates.
 4. Way too much of an academic approach to language design; appears to
 be a grand experiment to see how many academic papers can be derived
 from it.

Ok, academic research is bad.
 5. Newcomers to the language will find it's type system concepts
 overwhelming - co-variance and contra-variance etc.  (don't know how D2
 will address this better though). Yes these issues are important for OO
 libraries but feel there must be a more practical way out of the
 language complexity.  Personally I always kept away from the hairy and
 scary bits of C++; you don't need 'em in a practical language.

I know many professional Java/C# coders who know the concepts of variances. The concepts are not new.. have you ever written code in Java (? extends Foo) etc. ? Java is a simple language, much simpler than D and even in Java land you need to care about these. I have also heard C++ developers pondering these issues. Giving them the proper names comes from the academia, though.
 I've heard Scala's argument that all the complexity is hidden in the
 libraries so need to worry about it.  Unfortunately I don't believe her.
  I learn a lot about a language by studying the library code and expect
 it to be as easy to read and understand as mainline code.

It is. The core language does not have e.g. arrays or AAs. Library code is known to be much more complex than ordinary application code. I would not recommend studying a new language by reading the sources of libraries. It is like trying to learn quick-sort by reading the sources of JVM, ffmpeg or the Linux kernel. For instance there is the book 'Programming in Scala', which is much better a starting point as the language also has some theoretical prerequisites. Reading the library code gets easier after reading the book.
 5. Not her fault (i.e. of the language), but after six months of
 courting Scala with the Eclipse plugin, suffering IDE crash after crash
 and lost code I just could not bring myself to suffering her any longer.

When Descent was new, it also sucked. It used to hang badly when it parsed partial code, like when one did not yet have a closing bracket for arrays. The Netbeans plugin for Scala is more stable.
Sep 11 2009
parent Justin Johansson <procode adam-dott-com.au> writes:
 Ok, so you did not like optional semicolons.

 3. Half-baked embeddable XML support in the language looks like she
 borrowed from ECMAScript's E4X.


 4. Too many different ways of doing things.  All very interesting and no

for, foreach, foreach_reverse, do-while, while, recursive functions,

 4. Way too much of an academic approach to language design; appears to


Further, I'd much rather see universities teach Scala in CS courses in preference to any of C++, Java or D. The latter are all something one can learn on the job if they have a good founding in the fundamentals. Since Scala's designer studied under Nicholas Wirth, arguably Scala is the successor of Modula-2 and Pascal (two languages developed by Wirth) and it wouldn't be any surprise if Scala becomes the next teaching language. My original point was that Scala being in my opinion too academic, was not for my developer's taste; not that anything academic is bad.
 'Programming in Scala', which is much better a starting point as the 

 5. Not her fault (i.e. of the language), but after six months of
 courting Scala with the Eclipse plugin, suffering IDE crash after crash
 and lost code I just could not bring myself to suffering her any longer.

When Descent was new, it also sucked. It used to hang badly when it parsed partial code, like when one did not yet have a closing bracket for arrays. The Netbeans plugin for Scala is more stable.

Anyway, thanks for your comments on my comments .. all in the spirit of good debate. Cheers Justin
Sep 11 2009
prev sibling next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Justin Johansson:

I'm somewhat reluctant to discuss Scala too much here as this is a D forum,<

Discussing other languages is allowed here, especially if they show good things or bad things that may help D development.
 2. Scala cannot make up her mind if she's a scripting language or serious
language.<

Scala purpose is to try to be good for both small and large programs, that's the meaning of Scala, that stands for "Scalable language". I'd like to see D2 become a little fitter for small programs.
Optional semicolons at the end of statements are really frustrating for code
style consistency.<

Semicolons are noise, they slow down programming a little. Better to design future languages where newlines are enough. See also the Delight language, that's better than normal D.
 Read Cedric's blog June 2008 for example
 http://beust.com/weblog/archives/000490.html

The comments to that blog post are more intelligent and useful than the main post. See for example the comment by Amit Patel.
http://www.unlimitednovelty.com/2009/04/why-i-dont-like-scala.html<
Furthermore, Scala's object model is not only familiar, it's extremely
well-studied and well-optimized. The JVM provides immense capability to inline
method calls, which means calls which span multiple objects can be condensed
down to a single function call. This is because the Smalltalk-inspired illusion
that these objects are receiving and sending messages is completely suspended,
and objects are treated in C++-style as mere chunks of state, thus an inlined
method call can act on many of them at once as if they were simple chunks of
state. In Reia, all objects are concurrent, share no state, and can only
communicate with messages. Inlining calls across objects is thoroughly
impossible since sending messages in Reia is not some theoretical construct,
it's what really happens and cannot simply be abstracted away into a function
call which mutates the state of multiple objects.<

D compilers are generally not even able to inline most virtual calls, so Reia doesn't look like a fitting design for a language that must be fast as D. Reia design looks good for a higher level language. Scala is designed to be faster than Reia. Maybe someday people will find ways to efficiently compile Reia too, similar things have appened several times.
http://clojure.org/<

Clojure is not OOP, and it lacks several of the modern functional features (like a powerful type system). And it's slow. I don't like it a lot, despite its management of immutability is cute. Something about this topic: http://blog.higher-order.net/2009/02/01/understanding-clojures-persistentvector-implementation/ http://blog.higher-order.net/2009/09/08/understanding-clojures-persistenthashmap-deftwice/ Bye, bearophile
Sep 11 2009
next sibling parent reply Justin Johansson <procode adam-dott-com.au> writes:
bearophile Wrote:
 Justin Johansson:
I'm somewhat reluctant to discuss Scala too much here as this is a D forum,<


Okay. Thanks for mentioning that, bearophile, and your other reply comments noted. Since I'm still feeling D around the edges, would you mind saying what salient things there are about D that presumably attracts to the language. It just helps to know why others are here as one ventures into new territory. Cheers Justin Johansson
Sep 11 2009
next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Justin Johansson:

would you mind saying what salient things there are about D that presumably
attracts to the language.  It just helps to know why others are here as one
ventures into new territory.<

That's not an easy question. This is a personal answer, other people will like other sides of D. I like D1/D2 for: - I don't think of it as a propetary language, like C#. - Sometimes I want the freedom to use memory as I like, with structs, and even unions. If you have large datasets you find that using more than 20 bytes for a number as in Python doesn't help. Values also reduce indirection, this speeds up things. This allows a more efficient usage of memory, and this helps increase cache locality, that increases performance. Unfortunately GC-managed D pointers can't be tagged, so I have to use memory from the C heap for them. And sometimes you need pointers. That's why I'd like D to have more/better ways to increase safety when using pointers (like using memory regions when not in release mode, etc). - I like this newsgroups, I can talk to people, and they sometimes answer my numerous questions. I am learning a lot. Sometimes I receive no answers, but it's acceptable. For its nature this newsgroup attracts some strange people too. - I often use Python, it's the language I like more, but for some purposes it's too much slow. And I am tired of writing vectorized code in NumPy and the like. Cython reference count makes me sick and ShedSkin while nice is a failed experiment. D feels like freedom, while sometimes using Python feels like programming with mittens for me. - There are some things that I'd like to see in a language, and D being in development still and being not controlled by an ivory tower committee give me the illusion to see some of my ideas realized. So far I haven't influenced a lot the development of D. On the other hand if everyone can influence a lot the language the result may be a patchwork. So some dynamic compromise has to be found every day. - D looks a lot like C, yet in D I can write code several times faster than C. Sometimes 5-10 times faster. This is an enormous difference. - I am quite sensitive to syntax noise, boilerplate code. I like elegance, clarity in semantics, I hate corner cases, yet I want a language that's efficient, readable, and the less bug-prone as possible. C++ looks too much complex for me. D1 is simple enough for me, and D2 is getting a bit too much complex. I may appreciate the idea of a D 1.5 language that fixes some holes and limits of D1 while keeping language simple enough (struct constructors, and few other things. Such things don't significantly increase the difficulty in using the language). - I like how D doesn't totally ignore safety as C does, in D sometimes the default way is the safer one, and the unsafe way is used only where you ask for it. I'd like to see more safeties added to D, like optional run-time and compile-time integral overflow tests, some pointer safety, better template error messages (template constraints help some in such regard), stack traces, less compiler bugs, safer casts (in C# you need explicit casts to convert double => float), a safer printf, some optional annotations inspired by Plint (a lint program) to give more semantics to the compiler, that can be used to both speed up code and avoid bugs. There's lot that can be done in this regard. And release-mode performance can be usually kept unchanged. - I like how templates allow me to do some things that are doable only in less common languages like Lisp and Haskell, or in dynamic languages. - To use D I don't need to fill forms, receive emails, pay, install with an installer, or use an IDE. The compiler is free, you just need to download a zip, and recently such zip is well organized too inside. I uncompress the zip, seth a path or two and I am already able to use it. - The LDC compiler on Linux produces binaries that are efficient almost as C++ and sometimes more. Someday LDC will be available on Windows too. LDC developers are good people, they fix things very quickly (often in 24 hours), and they don't ignore user requests. For example in LDC the == among associative arrays now works. LDC developers are almost as serious people as LLVM devs, but no one gets paid for LDC (while the head of LLVM is paid by Apple). - D contains many tricks and handy features that save lot of time and make programming shorter and simpler. Some of such features are half-unfinished (some GC/GC-pointers semantics, module system, unittest system, contract programming, and several other things) and Walter doesn't seem willing to finish them soon, but having them partially unfinished is better than not having them. So in summary I like the freedom D gives me in using memory, and the freedom to program in the style I see most fit for a program, its work-in-progress nature that gives the illusion to be able to influence it in good ways, its simple enough nature, its handy features, its clear and short enough syntax (even if there are ways to improve it still). I like people in the D community, because they sometimes understand what's the way to improve the language. But probably the top two reasons for me are that it allows me to write fast programs and at the same time it's not a very bug-prone language. Helping me avoid bugs saves a ton of my programming time. Saves my time both when I program and when I run the program. Compared to this all other things are secondary. Bye, bearophile
Sep 11 2009
next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Saves my time both when I program and when I run the program. Compared to this
all other things are secondary.<

"Saving time" is one of the most important qualities of a programming language. A language can save time in many ways: - If it's easy to find with Google, if it's fast to download and install. If you have to pay for it you need waste some time. Sometimes installers help save time, but a zip file too can sometimes save time compared to an ugly installer. If you don't have to fill forms to download it it saves your time. Having an integrated editor, or having a IDE+compiler can save programming time. A good IDE can even turn a boring language like Java in an usable and quite useful one. Languages like C# are designed to almost require an IDE. - Having good online documentation and a good help can save lot of time. - A rich, easy to use and well debugged standard library can save a lot of time. Having a community of code (like Python modules you can find online to do almost everything, that are written in an uniform style that's easy to read and understand) can save a very large amount of time, even months. - If it's simple or similar to other languages, if its semantics is clear, this saves some time to learn it, sometimes many months. - A compact syntax saves a little programming time. A clear syntax improves readability, and this saves a lot of time when the program has to be debugged, modified, improved or just undertood. - A clear and high level semantics allows the programmer to think less about details, and this speeds up the invention or implementation of algorithms, and saves time (Python is among the best at this).. - If it helps avoid bugs or remove them it can save lot of programming time. - Some built-in features of the language can save some programming time. - If the compilation process is simple this saves time (see the Bud tool). If the language is designed to be compiled quickly (or to not require compilation) this saves time. - If programs run quickly it saves some time. A good language has to try to save time in all those ways and more. Bye, bearophile
Sep 11 2009
parent Justin Johansson <procode adam-dott-com.au> writes:
 - If programs run quickly it saves some time.
 
 A good language has to try to save time in all those ways and more.

Tks bearophile for that extensive writeup. A good read. btw. Downloaded the Bud tool (on linux) but couldn't get it to compile. First had to rename usage of "macro" to "makro" then it bitched about some missing module so I gave up. -- Justin
Sep 13 2009
prev sibling parent reply Jeremie Pelletier <jeremiep gmail.com> writes:
bearophile Wrote:

 Justin Johansson:
 
would you mind saying what salient things there are about D that presumably
attracts to the language.  It just helps to know why others are here as one
ventures into new territory.<

That's not an easy question. This is a personal answer, other people will like other sides of D. I like D1/D2 for: - I don't think of it as a propetary language, like C#. - Sometimes I want the freedom to use memory as I like, with structs, and even unions. If you have large datasets you find that using more than 20 bytes for a number as in Python doesn't help. Values also reduce indirection, this speeds up things. This allows a more efficient usage of memory, and this helps increase cache locality, that increases performance. Unfortunately GC-managed D pointers can't be tagged, so I have to use memory from the C heap for them. And sometimes you need pointers. That's why I'd like D to have more/better ways to increase safety when using pointers (like using memory regions when not in release mode, etc).

I haven't had to use the C heap whatsoever so far in D, could you give me an example of where you need it? In fact, the *only* place I use the C heap is in my garbage collector's internals, for pool structs and mark ranges. I use pointers to GC memory all the time too, there are plenty of algorithms, especially in loops, that can run faster with pointer arithmetic than slices and it's still the fastest way to pass struct references around.
 - I like this newsgroups, I can talk to people, and they sometimes answer my
numerous questions. I am learning a lot. Sometimes I receive no answers, but
it's acceptable. For its nature this newsgroup attracts some strange people too.
 - I often use Python, it's the language I like more, but for some purposes
it's too much slow. And I am tired of writing vectorized code in NumPy and the
like. Cython reference count makes me sick and ShedSkin while nice is a failed
experiment. D feels like freedom, while sometimes using Python feels like
programming with mittens for me.
 - There are some things that I'd like to see in a language, and D being in
development still and being not controlled by an ivory tower committee give me
the illusion to see some of my ideas realized. So far I haven't influenced a
lot the development of D. On the other hand if everyone can influence a lot the
language the result may be a patchwork. So some dynamic compromise has to be
found every day.

I also like this community driven model, but this forum has more people submitting ideas than people able to implement them on time, I'm pretty sure the TODO list is rather huge at this time :) I for one much prefer D development the way it is now than the "working group" model used by the W3C or Khronos for example. The public bugzilla is really nice too, once you get used to it, one of the issues I submitted got fixed in 2.032, I've also sent maybe 3 or 4 patches to the compiler source in other issues so far too, hopefully they'll be used in 2.033!
 - D looks a lot like C, yet in D I can write code several times faster than C.
Sometimes 5-10 times faster. This is an enormous difference.

Indeed, and sometimes it's way faster than that. I've ported many C headers to D and I'm always amazed at how many things I can throw out, just the DirectX headers were at least 50% smaller in D and MUCH easier to read. Such simplicity is also reflected in the compiler by having quite a lot less tokens and parse nodes to create and analyze. I must admit however that I sometimes miss the C preprocessor, or at least wish mixins had a syntax closer to that used by the C preprocessor. But it's a good idea to keep D without a preprocessor, its much better for everything to have a scope.
 - I am quite sensitive to syntax noise, boilerplate code. I like elegance,
clarity in semantics, I hate corner cases, yet I want a language that's
efficient, readable, and the less bug-prone as possible. C++ looks too much
complex for me. D1 is simple enough for me, and D2 is getting a bit too much
complex. I may appreciate the idea of a D 1.5 language that fixes some holes
and limits of D1 while keeping language simple enough (struct constructors, and
few other things. Such things don't significantly increase the difficulty in
using the language).

C++ isn't anymore complex than D2, its syntax just isn't as elegant. Other than multiple inheritance which is partially solved through object composition, I can't think of many features C++ has over D2. I can name quite a few features D2 has over C++ :) What I like about D is that while its elegant, it still allows for dirty work to be done when you need it. You don't need to code your application core in C and your application behavior in a scripting language on top of the C core. D allows you to write it all in one language with the same productivity, if not better productivity for not having to write the abstraction layer between C and scripting.
 - I like how D doesn't totally ignore safety as C does, in D sometimes the
default way is the safer one, and the unsafe way is used only where you ask for
it.  I'd like to see more safeties added to D, like optional run-time and
compile-time integral overflow tests, some pointer safety, better template
error messages (template constraints help some in such regard), stack traces,
less compiler bugs, safer casts (in C# you need explicit casts to convert
double => float), a safer printf, some optional annotations inspired by Plint
(a lint program) to give more semantics to the compiler, that can be used to
both speed up code and avoid bugs. There's lot that can be done in this regard.
And release-mode performance can be usually kept unchanged.

Stack traces is a feature for the runtime, I made one for mine, which shows a dialog window with the stack trace, current registers values, loaded modules and whatnot. Took me almost a week to piece together my CodeView reader, I still need a DWARF reader for linux. I'm gonna try and implement it in druntime and submit the patch to bugzilla.
 - I like how templates allow me to do some things that are doable only in less
common languages like Lisp and Haskell, or in dynamic languages.
 - To use D I don't need to fill forms, receive emails, pay, install with an
installer, or use an IDE. The compiler is free, you just need to download a
zip, and recently such zip is well organized too inside. I uncompress the zip,
seth a path or two and I am already able to use it.
 - The LDC compiler on Linux produces binaries that are efficient almost as C++
and sometimes more. Someday LDC will be available on Windows too. LDC
developers are good people, they fix things very quickly (often in 24 hours),
and they don't ignore user requests. For example in LDC the == among
associative arrays now works. LDC developers are almost as serious people as
LLVM devs, but no one gets paid for LDC (while the head of LLVM is paid by
Apple).
 - D contains many tricks and handy features that save lot of time and make
programming shorter and simpler. Some of such features are half-unfinished
(some GC/GC-pointers semantics, module system, unittest system, contract
programming, and several other things) and Walter doesn't seem willing to
finish them soon, but having them partially unfinished is better than not
having them.

There are hundreds of things on Walter's list, he can't magically do them all overnight :)
 
 So in summary I like the freedom D gives me in using memory, and the freedom
to program in the style I see most fit for a program, its work-in-progress
nature that gives the illusion to be able to influence it in good ways, its
simple enough nature, its handy features, its clear and short enough syntax
(even if there are ways to improve it still). I like people in the D community,
because they sometimes understand what's the way to improve the language.
 
 But probably the top two reasons for me are that it allows me to write fast
programs and at the same time it's not a very bug-prone language. Helping me
avoid bugs saves a ton of my programming time. Saves my time both when I
program and when I run the program. Compared to this all other things are
secondary.
 
 Bye,
 bearophile

Sep 13 2009
next sibling parent reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Jeremie Pelletier wrote:
 I haven't had to use the C heap whatsoever so far in D, could you give me an
example of where you need it? In fact, the *only* place I use the C heap is in
my garbage collector's internals, for pool structs and mark ranges. I use
pointers to GC memory all the time too, there are plenty of algorithms,
especially in loops, that can run faster with pointer arithmetic than slices
and it's still the fastest way to pass struct references around.

I use the C heap a lot when I need slabs of memory that the GC should not look into for performance reasons. This includes images/textures, mesh data and some data structures that I release manually - again, for efficiency reasons.
 - I like how D doesn't totally ignore safety as C does, in D sometimes the
default way is the safer one, and the unsafe way is used only where you ask for
it.  I'd like to see more safeties added to D, like optional run-time and
compile-time integral overflow tests, some pointer safety, better template
error messages (template constraints help some in such regard), stack traces,
less compiler bugs, safer casts (in C# you need explicit casts to convert
double => float), a safer printf, some optional annotations inspired by Plint
(a lint program) to give more semantics to the compiler, that can be used to
both speed up code and avoid bugs. There's lot that can be done in this regard.
And release-mode performance can be usually kept unchanged.

Stack traces is a feature for the runtime, I made one for mine, which shows a dialog window with the stack trace, current registers values, loaded modules and whatnot. Took me almost a week to piece together my CodeView reader, I still need a DWARF reader for linux. I'm gonna try and implement it in druntime and submit the patch to bugzilla.

Tango's runtime already does stack tracing on Windows and *NIX, however its CV parser is subject to some licensing issues :( Perhaps you could release yours under some liberal license so it can be plugged there? :) -- Tomasz Stachowiak http://h3.team0xf.com/ h3/h3r3tic on #D freenode
Sep 13 2009
parent reply Jeremie Pelletier <jeremiep gmail.com> writes:
Tom S Wrote:

 Jeremie Pelletier wrote:
 I haven't had to use the C heap whatsoever so far in D, could you give me an
example of where you need it? In fact, the *only* place I use the C heap is in
my garbage collector's internals, for pool structs and mark ranges. I use
pointers to GC memory all the time too, there are plenty of algorithms,
especially in loops, that can run faster with pointer arithmetic than slices
and it's still the fastest way to pass struct references around.

I use the C heap a lot when I need slabs of memory that the GC should not look into for performance reasons. This includes images/textures, mesh data and some data structures that I release manually - again, for efficiency reasons.

The garbage collector in D already mark allocations which contains pointers and scans these only. If you want to know if a type contains pointers, check the 'flags' property of the typeinfo or classinfo, test for bit0 and bit1 respectively. This is what the GC uses at runtime when allocating memory to know if it should tag the allocation as containing possible pointers. I myself allocate all my meshes and textures directly on the GC and I'm pretty sure its faster than C's malloc and much safer.
 - I like how D doesn't totally ignore safety as C does, in D sometimes the
default way is the safer one, and the unsafe way is used only where you ask for
it.  I'd like to see more safeties added to D, like optional run-time and
compile-time integral overflow tests, some pointer safety, better template
error messages (template constraints help some in such regard), stack traces,
less compiler bugs, safer casts (in C# you need explicit casts to convert
double => float), a safer printf, some optional annotations inspired by Plint
(a lint program) to give more semantics to the compiler, that can be used to
both speed up code and avoid bugs. There's lot that can be done in this regard.
And release-mode performance can be usually kept unchanged.

Stack traces is a feature for the runtime, I made one for mine, which shows a dialog window with the stack trace, current registers values, loaded modules and whatnot. Took me almost a week to piece together my CodeView reader, I still need a DWARF reader for linux. I'm gonna try and implement it in druntime and submit the patch to bugzilla.

Tango's runtime already does stack tracing on Windows and *NIX, however its CV parser is subject to some licensing issues :( Perhaps you could release yours under some liberal license so it can be plugged there? :)

Sure, I wouldn't mind at all, I'm not into licenses myself so I might just release it to public domain. I'll try and get a standalone package ready and post it somewhere, I just don't know where yet :x
Sep 13 2009
parent reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Jeremie Pelletier wrote:
 Tom S Wrote:
 
 Jeremie Pelletier wrote:
 I haven't had to use the C heap whatsoever so far in D, could you give me an
example of where you need it? In fact, the *only* place I use the C heap is in
my garbage collector's internals, for pool structs and mark ranges. I use
pointers to GC memory all the time too, there are plenty of algorithms,
especially in loops, that can run faster with pointer arithmetic than slices
and it's still the fastest way to pass struct references around.

not look into for performance reasons. This includes images/textures, mesh data and some data structures that I release manually - again, for efficiency reasons.

The garbage collector in D already mark allocations which contains pointers and scans these only. If you want to know if a type contains pointers, check the 'flags' property of the typeinfo or classinfo, test for bit0 and bit1 respectively. This is what the GC uses at runtime when allocating memory to know if it should tag the allocation as containing possible pointers.

Yea I know, but I want structures with pointers manually managed as well.
 I myself allocate all my meshes and textures directly on the GC and I'm pretty
sure its faster than C's malloc and much safer.

Hm, why would it be faster with the GC than malloc? I'm pretty sure it's the opposite :P Plus, I could use a specialized malloc implementation, like TLSF.
 
 - I like how D doesn't totally ignore safety as C does, in D sometimes the
default way is the safer one, and the unsafe way is used only where you ask for
it.  I'd like to see more safeties added to D, like optional run-time and
compile-time integral overflow tests, some pointer safety, better template
error messages (template constraints help some in such regard), stack traces,
less compiler bugs, safer casts (in C# you need explicit casts to convert
double => float), a safer printf, some optional annotations inspired by Plint
(a lint program) to give more semantics to the compiler, that can be used to
both speed up code and avoid bugs. There's lot that can be done in this regard.
And release-mode performance can be usually kept unchanged.


its CV parser is subject to some licensing issues :( Perhaps you could release yours under some liberal license so it can be plugged there? :)

Sure, I wouldn't mind at all, I'm not into licenses myself so I might just release it to public domain. I'll try and get a standalone package ready and post it somewhere, I just don't know where yet :x

Sweet :D As for a place, there are plenty of options, e.g. http://dsource.org/projects/scrapple/ or a separate dsource project. -- Tomasz Stachowiak http://h3.team0xf.com/ h3/h3r3tic on #D freenode
Sep 13 2009
parent reply Jeremie Pelletier <jeremiep gmail.com> writes:
Tom S Wrote:

 Jeremie Pelletier wrote:
 Tom S Wrote:
 
 Jeremie Pelletier wrote:
 I haven't had to use the C heap whatsoever so far in D, could you give me an
example of where you need it? In fact, the *only* place I use the C heap is in
my garbage collector's internals, for pool structs and mark ranges. I use
pointers to GC memory all the time too, there are plenty of algorithms,
especially in loops, that can run faster with pointer arithmetic than slices
and it's still the fastest way to pass struct references around.

not look into for performance reasons. This includes images/textures, mesh data and some data structures that I release manually - again, for efficiency reasons.

The garbage collector in D already mark allocations which contains pointers and scans these only. If you want to know if a type contains pointers, check the 'flags' property of the typeinfo or classinfo, test for bit0 and bit1 respectively. This is what the GC uses at runtime when allocating memory to know if it should tag the allocation as containing possible pointers.

Yea I know, but I want structures with pointers manually managed as well.

Then just inform the GC to not scan the allocations you want, or better yet have a static ctor modify the flag of the typeinfo you don't want scanned.
 I myself allocate all my meshes and textures directly on the GC and I'm pretty
sure its faster than C's malloc and much safer.

Hm, why would it be faster with the GC than malloc? I'm pretty sure it's the opposite :P Plus, I could use a specialized malloc implementation, like TLSF.

The D GC is already specialized, and given its being used quite a lot in D, there are good chances its already sitting in the CPU cache, its heap already having the available memory block waiting on a freelist, or if the alloc is more than 0x1000 bytes, the pages available in a pool. You'd need to use malloc quite a lot to get the same optimal performance, and mixing the two would affect the performance of both.
 - I like how D doesn't totally ignore safety as C does, in D sometimes the
default way is the safer one, and the unsafe way is used only where you ask for
it.  I'd like to see more safeties added to D, like optional run-time and
compile-time integral overflow tests, some pointer safety, better template
error messages (template constraints help some in such regard), stack traces,
less compiler bugs, safer casts (in C# you need explicit casts to convert
double => float), a safer printf, some optional annotations inspired by Plint
(a lint program) to give more semantics to the compiler, that can be used to
both speed up code and avoid bugs. There's lot that can be done in this regard.
And release-mode performance can be usually kept unchanged.


its CV parser is subject to some licensing issues :( Perhaps you could release yours under some liberal license so it can be plugged there? :)

Sure, I wouldn't mind at all, I'm not into licenses myself so I might just release it to public domain. I'll try and get a standalone package ready and post it somewhere, I just don't know where yet :x

Sweet :D As for a place, there are plenty of options, e.g. http://dsource.org/projects/scrapple/ or a separate dsource project.

I thought of that, but I don't feel like opening a project for just a few random code snippets or standalone classes. I think I'll just post it in this forum and let interested people grab it for now.
Sep 13 2009
next sibling parent reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Jeremie Pelletier wrote:
 Tom S Wrote:
 
 Jeremie Pelletier wrote:
 I myself allocate all my meshes and textures directly on the GC and I'm pretty
sure its faster than C's malloc and much safer.

the opposite :P Plus, I could use a specialized malloc implementation, like TLSF.

The D GC is already specialized, and given its being used quite a lot in D, there are good chances its already sitting in the CPU cache, its heap already having the available memory block waiting on a freelist, or if the alloc is more than 0x1000 bytes, the pages available in a pool. You'd need to use malloc quite a lot to get the same optimal performance, and mixing the two would affect the performance of both.

It might be specialized for _something_, but it definitely isn't real-time systems. I'd say with my use cases there's a very poor chance the GC is sitting in the CPU cache since most of the time my memory is preallocated and managed by specialized structures and/or malloc. I've found that using the GC only for the hard-to-manually-manage objects works best. The rest is handled by malloc and the GC has a very shallow vision of the world thus its collection runs are very fast. Of course there's a drawback that both the GC and malloc will have some pages cached, wasting memory, but I don't let the GC touch too much so it should be minimal. YMMV of course - all depends on the memory allocation patterns of the application.
 - I like how D doesn't totally ignore safety as C does, in D sometimes the
default way is the safer one, and the unsafe way is used only where you ask for
it.  I'd like to see more safeties added to D, like optional run-time and
compile-time integral overflow tests, some pointer safety, better template
error messages (template constraints help some in such regard), stack traces,
less compiler bugs, safer casts (in C# you need explicit casts to convert
double => float), a safer printf, some optional annotations inspired by Plint
(a lint program) to give more semantics to the compiler, that can be used to
both speed up code and avoid bugs. There's lot that can be done in this regard.
And release-mode performance can be usually kept unchanged.


its CV parser is subject to some licensing issues :( Perhaps you could release yours under some liberal license so it can be plugged there? :)


http://dsource.org/projects/scrapple/ or a separate dsource project.

I thought of that, but I don't feel like opening a project for just a few random code snippets or standalone classes. I think I'll just post it in this forum and let interested people grab it for now.

WORKSFORME :) -- Tomasz Stachowiak http://h3.team0xf.com/ h3/h3r3tic on #D freenode
Sep 13 2009
parent reply Jeremie Pelletier <jeremiep gmail.com> writes:
Tom S Wrote:

 Jeremie Pelletier wrote:
 Tom S Wrote:
 
 Jeremie Pelletier wrote:
 I myself allocate all my meshes and textures directly on the GC and I'm pretty
sure its faster than C's malloc and much safer.

the opposite :P Plus, I could use a specialized malloc implementation, like TLSF.

The D GC is already specialized, and given its being used quite a lot in D, there are good chances its already sitting in the CPU cache, its heap already having the available memory block waiting on a freelist, or if the alloc is more than 0x1000 bytes, the pages available in a pool. You'd need to use malloc quite a lot to get the same optimal performance, and mixing the two would affect the performance of both.

It might be specialized for _something_, but it definitely isn't real-time systems. I'd say with my use cases there's a very poor chance the GC is sitting in the CPU cache since most of the time my memory is preallocated and managed by specialized structures and/or malloc. I've found that using the GC only for the hard-to-manually-manage objects works best. The rest is handled by malloc and the GC has a very shallow vision of the world thus its collection runs are very fast. Of course there's a drawback that both the GC and malloc will have some pages cached, wasting memory, but I don't let the GC touch too much so it should be minimal. YMMV of course - all depends on the memory allocation patterns of the application.

I understand your points for using a separate memory manager, and I agree with you that having less active allocations make for faster sweeps, no matter how little of them are scanned for pointers. However I just had an idea on how to implement generational collection on a non-moving GC which should solve your issues (and well, mines too) with the collector not being fast enough. I need to do some hacking on my custom GC first, but I believe it could give yet another performance boost. I'll add my memory manager to my list of code modules to make public :)
Sep 13 2009
next sibling parent Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Jeremie Pelletier wrote:
 Tom S Wrote:
 
 Jeremie Pelletier wrote:
 Tom S Wrote:

 Jeremie Pelletier wrote:
 I myself allocate all my meshes and textures directly on the GC and I'm pretty
sure its faster than C's malloc and much safer.

the opposite :P Plus, I could use a specialized malloc implementation, like TLSF.


real-time systems. I'd say with my use cases there's a very poor chance the GC is sitting in the CPU cache since most of the time my memory is preallocated and managed by specialized structures and/or malloc. I've found that using the GC only for the hard-to-manually-manage objects works best. The rest is handled by malloc and the GC has a very shallow vision of the world thus its collection runs are very fast. Of course there's a drawback that both the GC and malloc will have some pages cached, wasting memory, but I don't let the GC touch too much so it should be minimal. YMMV of course - all depends on the memory allocation patterns of the application.

I understand your points for using a separate memory manager, and I agree with you that having less active allocations make for faster sweeps, no matter how little of them are scanned for pointers. However I just had an idea on how to implement generational collection on a non-moving GC which should solve your issues (and well, mines too) with the collector not being fast enough. I need to do some hacking on my custom GC first, but I believe it could give yet another performance boost. I'll add my memory manager to my list of code modules to make public :)

Sounds great, I can't wait! :D -- Tomasz Stachowiak http://h3.team0xf.com/ h3/h3r3tic on #D freenode
Sep 13 2009
prev sibling next sibling parent reply Nick B <nickB gmail.com> writes:
Jeremie Pelletier wrote:
 Tom S Wrote:
 
 Jeremie Pelletier wrote:
 Tom S Wrote:

 Jeremie Pelletier wrote:
 I myself allocate all my meshes and textures directly on the GC and I'm pretty
sure its faster than C's malloc and much safer.

the opposite :P Plus, I could use a specialized malloc implementation, like TLSF.


real-time systems. I'd say with my use cases there's a very poor chance the GC is sitting in the CPU cache since most of the time my memory is preallocated and managed by specialized structures and/or malloc. I've found that using the GC only for the hard-to-manually-manage objects works best. The rest is handled by malloc and the GC has a very shallow vision of the world thus its collection runs are very fast. Of course there's a drawback that both the GC and malloc will have some pages cached, wasting memory, but I don't let the GC touch too much so it should be minimal. YMMV of course - all depends on the memory allocation patterns of the application.

I understand your points for using a separate memory manager, and I agree with you that having less active allocations make for faster sweeps, no matter how little of them are scanned for pointers. However I just had an idea on how to implement generational collection on a non-moving GC which should solve your issues (and well, mines too) with the collector not being fast enough. I need to do some hacking on my custom GC first, but I believe it could give yet another performance boost. I'll add my memory manager to my list of code modules to make public :)

If the code is really usefull, why not offer it to the Tango team, for formal inclusion in the next release ? Nick B
Sep 13 2009
parent Jeremie Pelletier <jeremiep gmail.com> writes:
Nick B Wrote:

 Jeremie Pelletier wrote:
 Tom S Wrote:
 
 Jeremie Pelletier wrote:
 Tom S Wrote:

 Jeremie Pelletier wrote:
 I myself allocate all my meshes and textures directly on the GC and I'm pretty
sure its faster than C's malloc and much safer.

the opposite :P Plus, I could use a specialized malloc implementation, like TLSF.


real-time systems. I'd say with my use cases there's a very poor chance the GC is sitting in the CPU cache since most of the time my memory is preallocated and managed by specialized structures and/or malloc. I've found that using the GC only for the hard-to-manually-manage objects works best. The rest is handled by malloc and the GC has a very shallow vision of the world thus its collection runs are very fast. Of course there's a drawback that both the GC and malloc will have some pages cached, wasting memory, but I don't let the GC touch too much so it should be minimal. YMMV of course - all depends on the memory allocation patterns of the application.

I understand your points for using a separate memory manager, and I agree with you that having less active allocations make for faster sweeps, no matter how little of them are scanned for pointers. However I just had an idea on how to implement generational collection on a non-moving GC which should solve your issues (and well, mines too) with the collector not being fast enough. I need to do some hacking on my custom GC first, but I believe it could give yet another performance boost. I'll add my memory manager to my list of code modules to make public :)

If the code is really usefull, why not offer it to the Tango team, for formal inclusion in the next release ? Nick B

Because I dropped support for D1 long ago. If either the Tango or Phobos team like my code once I publish it, they are free to adapt it for their runtime. I rewrote the GC from scratch and optimized over the past 2 years to support my custom D runtime. It cannot be used as-is with neither phobos or tango without either changing the public interface of the GC or rewriting every runtime routine calling into the GC. I would only release it to public domain as an example of how to implement a tracing generational non-moving GC. I still need to implement the generational part, but I got the general algorithm down on paper today so I should have it working sometime this week. I'm not a big fan of code licenses and therefore like to write most of my code myself, if only to learn how it works. I rarely mind people asking for my code either, so long as I get credited for it :)
Sep 14 2009
prev sibling parent reply Leandro Lucarella <llucax gmail.com> writes:
Jeremie Pelletier, el 13 de septiembre a las 22:58 me escribiste:
 Tom S Wrote:
 
 Jeremie Pelletier wrote:
 Tom S Wrote:
 
 Jeremie Pelletier wrote:
 I myself allocate all my meshes and textures directly on the GC and I'm pretty
sure its faster than C's malloc and much safer.

the opposite :P Plus, I could use a specialized malloc implementation, like TLSF.

The D GC is already specialized, and given its being used quite a lot in D, there are good chances its already sitting in the CPU cache, its heap already having the available memory block waiting on a freelist, or if the alloc is more than 0x1000 bytes, the pages available in a pool. You'd need to use malloc quite a lot to get the same optimal performance, and mixing the two would affect the performance of both.

It might be specialized for _something_, but it definitely isn't real-time systems. I'd say with my use cases there's a very poor chance the GC is sitting in the CPU cache since most of the time my memory is preallocated and managed by specialized structures and/or malloc. I've found that using the GC only for the hard-to-manually-manage objects works best. The rest is handled by malloc and the GC has a very shallow vision of the world thus its collection runs are very fast. Of course there's a drawback that both the GC and malloc will have some pages cached, wasting memory, but I don't let the GC touch too much so it should be minimal. YMMV of course - all depends on the memory allocation patterns of the application.

I understand your points for using a separate memory manager, and I agree with you that having less active allocations make for faster sweeps, no matter how little of them are scanned for pointers. However I just had an idea on how to implement generational collection on a non-moving GC which should solve your issues (and well, mines too) with the collector not being fast enough. I need to do some hacking on

I saw a paper about that. The idea was to simply have some list of objects/pages in each generation and modify that lists instead of moving objects. I can't remember the name of the paper so I can't find it now :S The problem with generational collectors (in D) is that you need read/write barriers to track inter-generational pointers (to be able to use pointers to younger generations in the older ones as roots when scanning), which can make the whole deal a little unpractical for a language that doesn't want to impose performance penalty to thing you wont use (I don't see a way to instrument read/writes to pointers to the GC only). This is why RC was always rejected as an algorithm for the GC in D, I think.
 my custom GC first, but I believe it could give yet another performance
 boost. I'll add my memory manager to my list of code modules to make
 public :)

-- Leandro Lucarella (luca) | Blog colectivo: http://www.mazziblog.com.ar/blog/ ---------------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ---------------------------------------------------------------------------- Pack and get dressed before your father hears us, before all hell breaks loose.
Sep 14 2009
parent reply Fawzi Mohamed <fmohamed mac.com> writes:
On 2009-09-14 17:07:00 +0200, "Robert Jacques" <sandford jhu.edu> said:

 On Mon, 14 Sep 2009 09:39:51 -0400, Leandro Lucarella 
 <llucax gmail.com>  wrote:
 Jeremie Pelletier, el 13 de septiembre a las 22:58 me escribiste:

 I understand your points for using a separate memory manager, and
 I agree with you that having less active allocations make for faster
 sweeps, no matter how little of them are scanned for pointers. However
 I just had an idea on how to implement generational collection on
 a non-moving GC which should solve your issues (and well, mines too)
 with the collector not being fast enough. I need to do some hacking on

I saw a paper about that. The idea was to simply have some list of objects/pages in each generation and modify that lists instead of moving objects. I can't remember the name of the paper so I can't find it now :S The problem with generational collectors (in D) is that you need read/write barriers to track inter-generational pointers (to be able to use pointers to younger generations in the older ones as roots when scanning), which can make the whole deal a little unpractical for a language that doesn't want to impose performance penalty to thing you wont use (I don't see a way to instrument read/writes to pointers to the GC only). This is why RC was always rejected as an algorithm for the GC in D, I think.
 my custom GC first, but I believe it could give yet another performance
 boost. I'll add my memory manager to my list of code modules to make
 public :)


As a counter-point, objective-c just introduced a thread-local GC. According to a blog post (http://www.sealiesoftware.com/blog/archive/2009/08/28/objc_explain_Thread-local_garb ge_collection.html) apparently this has allowed pause times similar to the pause times of the previous generational GC. (Except that the former is doing a full collect, and the later still has work to do) On that note, it would probably be a good idea if core.gc.BlkAttr supported shared and immutable state flags, which could be used to support a thread-local GC.

1) to allocate large objects that have a guard object it is a good idea to pass through the GC because if memory is tight a gc collection is triggered thereby possibly freeing some extra memory 2) using gc malloc is not faster than malloc, especially with several threads the single lock of the basic gc makes itself felt. for how I use D (not realtime) the two things I would like to see from new gc are: 1) multiple pools (at least one per cpu, with thread id hash to assign threads to a given pool). This to avoid the need of a global gc lock in the gc malloc, and if possible use memory close to the cpu when a thread is pinned, not to have really thread local memory, if you really need local memory different from the stack then maybe a separate process should be used. This is especially well doable with 64 bits, with 32 memory usage/fragmentation could become an issue. 2) multiple thread doing the collection (a main thread distributing the work to other threads (one per cpu), that do the mark phase using atomic ops). other better gc, less latency (but not at the cost of too much computation), would be nice to have, but are not a priority for my usage. Fawzi
Sep 14 2009
parent reply Fawzi Mohamed <fmohamed mac.com> writes:
On 2009-09-15 04:51:19 +0200, "Robert Jacques" <sandford jhu.edu> said:

 On Mon, 14 Sep 2009 18:53:51 -0400, Fawzi Mohamed <fmohamed mac.com> wrote:
 
 On 2009-09-14 17:07:00 +0200, "Robert Jacques" <sandford jhu.edu> said:
 
 On Mon, 14 Sep 2009 09:39:51 -0400, Leandro Lucarella  
 <llucax gmail.com>  wrote:
 Jeremie Pelletier, el 13 de septiembre a las 22:58 me escribiste:


idea to pass through the GC because if memory is tight a gc collection is triggered thereby possibly freeing some extra memory 2) using gc malloc is not faster than malloc, especially with several threads the single lock of the basic gc makes itself felt. for how I use D (not realtime) the two things I would like to see from new gc are: 1) multiple pools (at least one per cpu, with thread id hash to assign threads to a given pool). This to avoid the need of a global gc lock in the gc malloc, and if possible use memory close to the cpu when a thread is pinned, not to have really thread local memory, if you really need local memory different from the stack then maybe a separate process should be used. This is especially well doable with 64 bits, with 32 memory usage/fragmentation could become an issue. 2) multiple thread doing the collection (a main thread distributing the work to other threads (one per cpu), that do the mark phase using atomic ops). other better gc, less latency (but not at the cost of too much computation), would be nice to have, but are not a priority for my usage. Fawzi

For what it's worth, the whole point of thread-local GC is to do 1) and 2). For the purposes of clarity, thread-local GC refers to each thread having it's own GC for non-shared objects + a shared GC for shared objects. Each thread's GC may allocate and collect independently of each other (e.g. in parallel) without locking/atomics/etc.

Well I want at least thread local pools (or almost, one can probably restrict it to the number of cpus, which will give most of the benefit), but not an extra partition of the memory in thread local and shared. Such a partition might be easier in D2 (I think it was discussed, but even then I am not fully sure about the benefit), because then you have to somehow be able to share and maybe even unshare an object, which will be cumbersome. Thread local things add a level in the memory hierarchy that I am not fully sure is worth having, in it you should have almost only low level plumbing. If you really want that much separation for many things then maybe a separate process + memmap might be better. The fast local storage for me is the stack, and one might think about being more aggressive in using it, the heap is potentially shared. Well at least that is my feeling. Note that on 64 bit one can easily use a few bits to subdivide the memory in parts, making finding the pool group very quick, and this discussion is orthogonal to being generational or not. Fawzi
Sep 15 2009
parent reply Jeremie Pelletier <jeremiep gmail.com> writes:
Fawzi Mohamed Wrote:

 On 2009-09-15 04:51:19 +0200, "Robert Jacques" <sandford jhu.edu> said:
 
 On Mon, 14 Sep 2009 18:53:51 -0400, Fawzi Mohamed <fmohamed mac.com> wrote:
 
 On 2009-09-14 17:07:00 +0200, "Robert Jacques" <sandford jhu.edu> said:
 
 On Mon, 14 Sep 2009 09:39:51 -0400, Leandro Lucarella  
 <llucax gmail.com>  wrote:
 Jeremie Pelletier, el 13 de septiembre a las 22:58 me escribiste:


idea to pass through the GC because if memory is tight a gc collection is triggered thereby possibly freeing some extra memory 2) using gc malloc is not faster than malloc, especially with several threads the single lock of the basic gc makes itself felt. for how I use D (not realtime) the two things I would like to see from new gc are: 1) multiple pools (at least one per cpu, with thread id hash to assign threads to a given pool). This to avoid the need of a global gc lock in the gc malloc, and if possible use memory close to the cpu when a thread is pinned, not to have really thread local memory, if you really need local memory different from the stack then maybe a separate process should be used. This is especially well doable with 64 bits, with 32 memory usage/fragmentation could become an issue. 2) multiple thread doing the collection (a main thread distributing the work to other threads (one per cpu), that do the mark phase using atomic ops). other better gc, less latency (but not at the cost of too much computation), would be nice to have, but are not a priority for my usage. Fawzi

For what it's worth, the whole point of thread-local GC is to do 1) and 2). For the purposes of clarity, thread-local GC refers to each thread having it's own GC for non-shared objects + a shared GC for shared objects. Each thread's GC may allocate and collect independently of each other (e.g. in parallel) without locking/atomics/etc.

Well I want at least thread local pools (or almost, one can probably restrict it to the number of cpus, which will give most of the benefit), but not an extra partition of the memory in thread local and shared. Such a partition might be easier in D2 (I think it was discussed, but even then I am not fully sure about the benefit), because then you have to somehow be able to share and maybe even unshare an object, which will be cumbersome. Thread local things add a level in the memory hierarchy that I am not fully sure is worth having, in it you should have almost only low level plumbing. If you really want that much separation for many things then maybe a separate process + memmap might be better. The fast local storage for me is the stack, and one might think about being more aggressive in using it, the heap is potentially shared. Well at least that is my feeling. Note that on 64 bit one can easily use a few bits to subdivide the memory in parts, making finding the pool group very quick, and this discussion is orthogonal to being generational or not. Fawzi

I just posted my memory manager to pastebin: http://pastebin.com/f7459ba9d I gave up on the generational feature, its indeed impossible without write barriers to keep track of pointers from old generations to newer ones. I had the whole tracing algorithm done but without generations, a naive scan and sweep is faster because it has way less cache misses. I'd like to get some feedback on it if possible.
Sep 16 2009
parent reply Lutger <lutger.blijdestijn gmail.com> writes:
Jeremie Pelletier wrote:
...
 
 I just posted my memory manager to pastebin:
 http://pastebin.com/f7459ba9d
 
 I gave up on the generational feature, its indeed impossible without write
 barriers to keep track of pointers from old generations to newer ones. I
 had the whole tracing algorithm done but without generations, a naive scan
 and sweep is faster because it has way less cache misses.
 
 I'd like to get some feedback on it if possible.

I think that it deserves a new thread...
Sep 16 2009
parent dsimcha <dsimcha yahoo.com> writes:
== Quote from Lutger (lutger.blijdestijn gmail.com)'s article
 Jeremie Pelletier wrote:
 ...
 I just posted my memory manager to pastebin:
 http://pastebin.com/f7459ba9d

 I gave up on the generational feature, its indeed impossible without write
 barriers to keep track of pointers from old generations to newer ones. I
 had the whole tracing algorithm done but without generations, a naive scan
 and sweep is faster because it has way less cache misses.

 I'd like to get some feedback on it if possible.


Yes, preferably on D.announce, and please explain what you did for the people who didn't read the original (horribly long, off-topic) thread.
Sep 16 2009
prev sibling parent reply Lutger <lutger.blijdestijn gmail.com> writes:
Jeremie Pelletier wrote:

 Tom S Wrote:

 Sweet :D As for a place, there are plenty of options, e.g.
 http://dsource.org/projects/scrapple/ or a separate dsource project.

I thought of that, but I don't feel like opening a project for just a few random code snippets or standalone classes. I think I'll just post it in this forum and let interested people grab it for now.

That's cool, but scrapple is exactly that: an assortment of small(ish) projects / pieces of code that otherwise don't warrant a full project. If you feel like putting it online, just ping BCS and I'm sure he'll give you access right away. Stacktrace is a feature highly desired by lots of people, you will many developers happy :)
Sep 13 2009
parent BCS <none anon.com> writes:
Hello Lutger,
 That's cool, but scrapple is exactly that: an assortment of small(ish)
 projects / pieces of code that otherwise don't warrant a full project.
 If you feel like putting it online, just ping BCS and I'm sure he'll
 give you access right away.

All I need is your dsource user name.
Sep 14 2009
prev sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Jeremie Pelletier:

I haven't had to use the C heap whatsoever so far in D, could you give me an
example of where you need it?<

1) I'm able to allocate a bigger single chunk of memory from the C heap, about 1.8 GB, while the GC heap of DMD on Windows allows only for a smaller chunk. In a program I've had to allocate a single chunk of memory. 2) I think the GC is faster at allocating small chunks of memory, while the C heap is faster at allocating large chunks. 3) GC-managed pointers have several restrictions (so much different that maybe I'd like them to be seen as a different type from the compiler, that requires a cast to be converted from/to C pointers. I don't know why this idea was not appreciated by D designers). One of them is that GC-managed pointers can't be tagged, you can't add one or two bits to GC-managed pointers, and such tags are useful when you want to implement certain data structures.
Indeed, and sometimes it's way faster than that.<

But lot of people will judge D against more modern languages like C#, Scala 8or Java) and not against C.
C++ isn't anymore complex than D2,<

I don't agree, see below.
I can't think of many features C++ has over D2. I can name quite a few features
D2 has over C++ :)<

Complexity and not-simplicity come from many things, like corner cases, rules against rules against rules, unsafe things that the compiler isn't able to catch in case of programmer errors, unnatural syntax, and not just from features. For example the Python 3+ language has many more features than pure C, yet Python3 is simpler than C :-)
it still allows for dirty work to be done when you need it.<

The less dirty you make it the better it will be when you try to mantain/debug your D code :-)
You don't need to code your application core in C and your application behavior
in a scripting language on top of the C core. D allows you to write it all in
one language with the same productivity, if not better productivity for not
having to write the abstraction layer between C and scripting.<

While D is quite better than C, that of yours is a dream. In practice D isn't dynamic and for certain purposes D is not close to the "productivity" of Python :-) (Even just because in Python you can find tons of modules and bindings already done). There are ways to improve D still for such purposes. D can be more scalable as Scala :-) Bye, bearophile
Sep 14 2009
next sibling parent reply Lutger <lutger.blijdestijn gmail.com> writes:
language_fan wrote:

 Mon, 14 Sep 2009 07:33:59 -0400, bearophile thusly wrote:
 
 But lot of people will judge D against more modern languages like C#,
 Scala or Java) and not against C.

Programmers often belong to three kinds of groups. First come the fans of traditionally weakly typed compiled languages (basic, c, c++). They have tried some "dynamic" or "academic" languages but did not like them. They fancy efficiency and close to metal feel. They think compilation to native code is the best way to produce programs, and think types should reflect the feature set of their cpu. They believe the syntax C uses was defined by their God. The second group started with interpreted languages built by amateurs (php, ruby, python, some game scripting language etc). They do not understand the meaning the types or compilation. They prefer writing short programs that usually seem to work. They hate formal specifications and proofs about program properties. They are usually writing simple web applications or some basic shareware utilies no one uses. They also hate trailing semicolons. The members of the last group have studied computer science and languages, in particular. They have found a pet academic language, typically a pure one, but paradigms may differ. In fact this is the group which uses something other than the hybrid object-oriented/procedural model. They appreciate a strong, orthogonal core language that scales cleanly. They are not scared of esoteric non-C-like syntax. They use languages that are not ready to take a step to the "real world" during the 70 next years.

That's a fancy way of saying that anyone who has not studied CS is a moron and therefore cannot understand what is good about languages, thus they lose any argument automatically. Am I right?
Sep 14 2009
next sibling parent Jeremie Pelletier <jeremiep gmail.com> writes:
Lutger Wrote:

 language_fan wrote:
 
 Mon, 14 Sep 2009 07:33:59 -0400, bearophile thusly wrote:
 
 But lot of people will judge D against more modern languages like C#,
 Scala or Java) and not against C.

Programmers often belong to three kinds of groups. First come the fans of traditionally weakly typed compiled languages (basic, c, c++). They have tried some "dynamic" or "academic" languages but did not like them. They fancy efficiency and close to metal feel. They think compilation to native code is the best way to produce programs, and think types should reflect the feature set of their cpu. They believe the syntax C uses was defined by their God. The second group started with interpreted languages built by amateurs (php, ruby, python, some game scripting language etc). They do not understand the meaning the types or compilation. They prefer writing short programs that usually seem to work. They hate formal specifications and proofs about program properties. They are usually writing simple web applications or some basic shareware utilies no one uses. They also hate trailing semicolons. The members of the last group have studied computer science and languages, in particular. They have found a pet academic language, typically a pure one, but paradigms may differ. In fact this is the group which uses something other than the hybrid object-oriented/procedural model. They appreciate a strong, orthogonal core language that scales cleanly. They are not scared of esoteric non-C-like syntax. They use languages that are not ready to take a step to the "real world" during the 70 next years.

That's a fancy way of saying that anyone who has not studied CS is a moron and therefore cannot understand what is good about languages, thus they lose any argument automatically. Am I right?

I dunno if that's what OP meant, but studying CS does not make you a reference in programming languages. I didn't even complete my first year of CS because I wasn't learning as fast as I wanted. School teaches you theory anyways, a job will teach you how to apply it in the real world. Anyone who can read and has the slightest interest in programming can learn the theory by themselves. As for the different classes of programmers, I think the OP pushed more the extremes than the general cases. I came across a series of articles by Eric Lippert a few weeks ago talking about the matter: http://blogs.msdn.com/ericlippert/archive/tags/Cargo+Cult+Programming/default.aspx
Sep 14 2009
prev sibling next sibling parent Don <nospam nospam.com> writes:
language_fan wrote:
 Tue, 15 Sep 2009 00:25:46 +0200, Lutger thusly wrote:
 
 That's a fancy way of saying that anyone who has not studied CS is a
 moron and therefore cannot understand what is good about languages, thus
 they lose any argument automatically. Am I right?

I just recommend learning basic concepts until terms like generational garbage collection, closure, register allocation, immutability, loop fusion, term rewriting, regular languages, type constructor, virtual constructor, and covariance do not scare you anymore. If something small like optional semicolons or some other syntactic nuance prevents you from finishing your job, how the heck are you supposed to build any real world programs? Just to put this to some perspective, syntax matters, but not much.

Nowadays you can easily write
 a tool that parses stuff written in language X and outputs it in pretty 
 printed form in language Y. This is what happens on .NET, for instance. 
 Most of the languages there are just syntactic skins for the same common 
 core language.

It sounds as though talking about VB.NET, which is a non-existent language (it's a parsing step ONLY). It's just C# with a different parse table, and exists only for marketing reasons (to disguise the fact that MS abandoned VB). I don't think you can conclude anything general from that.
Sep 15 2009
prev sibling next sibling parent Justin Johansson <procode adam-dott-com.au> writes:
language_fan Wrote:

 ... Nowadays you can easily write 
 a tool that parses stuff written in language X and outputs it in pretty 
 printed form in language Y. This is what happens on .NET, for instance. 
 Most of the languages there are just syntactic skins for the same common 
 core language.

Yes, well, Ted Neward pretty much makes this observation re LLVM saying "Holy frickin' crap. I think I'm in love." In case you missed it, here's the link http://blogs.tedneward.com/2008/02/24/Some+Interesting+Tidbits+About+LLVM.aspx <JJ/>
Sep 15 2009
prev sibling parent Lutger <lutger.blijdestijn gmail.com> writes:
language_fan wrote:

 Tue, 15 Sep 2009 00:25:46 +0200, Lutger thusly wrote:
 
 That's a fancy way of saying that anyone who has not studied CS is a
 moron and therefore cannot understand what is good about languages, thus
 they lose any argument automatically. Am I right?

I just recommend learning basic concepts until terms like generational garbage collection, closure, register allocation, immutability, loop fusion, term rewriting, regular languages, type constructor, virtual constructor, and covariance do not scare you anymore.

Right right, I don't disagree with that. It was more the 'ruby/python programmers make apps no-one uses using amateur tools | c-family users worship FOO and the rest are academics that use pure functional languages' part that tripped me up. You know, the majority of software isn't built by academics, NASA uses C mostly, etc. A little nuance wouldn't hurt here.
Sep 15 2009
prev sibling next sibling parent Christopher Wright <dhasenan gmail.com> writes:
language_fan wrote:
 In fact this is the group 
 which uses something other than the hybrid object-oriented/procedural 
 model.

Damn straight! They use a hybrid OO/procedural/functional model. Like D. Or C#. "Oh, but Prolog," you may say. And I admit, I've seen it used a couple times by academics. But I've seen similar languages used at my job, and we're not by any means an algorithms shop. I'm actually aware of very few languages that break the mold. There are toy languages like Befunge; there are solver-oriented languages like Prolog and Zimpl; and there are a couple oddities like METAFONT. What languages have you seen that are so innovative and different in paradigm?
Sep 14 2009
prev sibling parent Rainer Deyke <rainerd eldwood.com> writes:
language_fan wrote:
 The members of the last group have studied computer science and 
 languages, in particular. They have found a pet academic language, 
 typically a pure one, but paradigms may differ. In fact this is the group 
 which uses something other than the hybrid object-oriented/procedural 
 model. They appreciate a strong, orthogonal core language that scales 
 cleanly. They are not scared of esoteric non-C-like syntax. They use 
 languages that are not ready to take a step to the "real world" during 
 the 70 next years.

Of the three types, this comes closest to describing me. Yet, I am completely self-taught, and my preferred language is still C++. (I wouldn't call it my pet language. I loathe C++, I just haven't found a suitable replacement yet.) Stereotypes are dangerous. -- Rainer Deyke - rainerd eldwood.com
Sep 14 2009
prev sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Mon, 14 Sep 2009 09:39:51 -0400, Leandro Lucarella <llucax gmail.com>  
wrote:
 Jeremie Pelletier, el 13 de septiembre a las 22:58 me escribiste:

 I understand your points for using a separate memory manager, and
 I agree with you that having less active allocations make for faster
 sweeps, no matter how little of them are scanned for pointers. However
 I just had an idea on how to implement generational collection on
 a non-moving GC which should solve your issues (and well, mines too)
 with the collector not being fast enough. I need to do some hacking on

I saw a paper about that. The idea was to simply have some list of objects/pages in each generation and modify that lists instead of moving objects. I can't remember the name of the paper so I can't find it now :S The problem with generational collectors (in D) is that you need read/write barriers to track inter-generational pointers (to be able to use pointers to younger generations in the older ones as roots when scanning), which can make the whole deal a little unpractical for a language that doesn't want to impose performance penalty to thing you wont use (I don't see a way to instrument read/writes to pointers to the GC only). This is why RC was always rejected as an algorithm for the GC in D, I think.
 my custom GC first, but I believe it could give yet another performance
 boost. I'll add my memory manager to my list of code modules to make
 public :)


As a counter-point, objective-c just introduced a thread-local GC. According to a blog post (http://www.sealiesoftware.com/blog/archive/2009/08/28/objc_explain_Thread-local_garb ge_collection.html) apparently this has allowed pause times similar to the pause times of the previous generational GC. (Except that the former is doing a full collect, and the later still has work to do) On that note, it would probably be a good idea if core.gc.BlkAttr supported shared and immutable state flags, which could be used to support a thread-local GC.
Sep 14 2009
prev sibling next sibling parent reply Rainer Deyke <rainerd eldwood.com> writes:
Nick Sabalausky wrote:
 That's *very* programmer-dependent. It originally took took me all of about 
 a week to get used to semicolons after growing up on basic (and even then it 
 was a very very minor time sink), and now just it takes all of about a split 
 second to press that key. But, any time I use a language that doesn't allow 
 semicolon line endings, I keep sticking them in without even thinking about 
 it. Then the compiler complains, and I have to go back and fix it, and that 
 slows down programming more than just instinctively hitting a key.

If you're going to judge features on the basis of habit, then the best language is always the language you have been using for the longest time. I'm not entirely happy with the way Scala handles the division between statements - Scala's rules seem arbitrary and complex - but semicolons *are* noise, no matter how habitually I use them and how much time I waste removing them afterwards. My preferred rule is this: If two lines have the same indentation, they are separate statements. If the second line is indented further than the first line, the second line is a continuation of the statement started in the first line. Surprisingly, even Python (which already has significant indentation) doesn't use this simple and obvious rule. -- Rainer Deyke - rainerd eldwood.com
Sep 11 2009
parent reply Benji Smith <dlanguage benjismith.net> writes:
Rainer Deyke wrote:
 I'm not entirely happy with the way Scala handles the division between
 statements - Scala's rules seem arbitrary and complex - but semicolons
 *are* noise, no matter how habitually I use them and how much time I
 waste removing them afterwards.

I don't know anything about scala, but I've been working on an Actionscript compiler recently (the language is based on ECMAScript, so it's very much like JavaScript in this respect) and the optional semicolon rules are completely maddening. The ECMAScript spec basically says: virtual semicolons must be inserted at end-of-line whenever the non-insertion of semicolons would result in an erroneous parse. So there are really only three ways to handle it, and all of them are insane: 1) Treat the newline character as a token (rather than as skippable whitespace) and include that token as an optional construct in every single production where it can legally occur. This results in hundreds of optional semicolons throughout the grammar, and makes the whole thing a nightmare to read, but at least it still uses a one-pass CFG. CLASS := "class" NEWLINE? IDENTIFIER NEWLINE? "{" NEWLINE? ( MEMBER NEWLINE? )* "}" 2) Use lexical lookahead, dispatched from the parser. The tokenizer determines whether to treat a newline as a statement terminator based on the current parse state (are we in the middle of a parenthetized expression?) and the upcoming tokens on the next line. This is nasty because the grammar becomes context-sensitive and conflates lexical analysis with parsing. 2) Whenever the parser encounters an error, have it back up to the beginning of the previous production and insert a virtual semicolon into the token stream. Then try reparsing. Since there might be multiple newlines contained in a single multiline expression, it might take arbitrarily many rewrite attempts before reaching a correct parse. The thing about most compiler construction tools is that they don't allow interaction between the context-guided tokenization, and they're not designed for the creation of backup-and-retry processing, or the insertion of virtual tokens into the token stream. Ugly stuff. Anyhoo, I know this is waaaaaaay off topic. But I think any language designer including optional semicolons in their language desperately deserves a good swift punch in the teeth. --benji
Sep 11 2009
parent Rainer Deyke <rainerd eldwood.com> writes:
Benji Smith wrote:
 The ECMAScript spec basically says: virtual semicolons must be inserted
 at end-of-line whenever the non-insertion of semicolons would result in
 an erroneous parse.

And I don't like that rule, and I've proposed a sensible alternative, but...
 So there are really only three ways to handle it, and all of them are
 insane:

 
 1) Treat the newline character as a token (rather than as skippable
 whitespace) and include that token as an optional construct in every
 single production where it can legally occur. This results in hundreds
 of optional semicolons throughout the grammar, and makes the whole thing
 a nightmare to read, but at least it still uses a one-pass CFG.
 
     CLASS :=
       "class"
       NEWLINE?
       IDENTIFIER
       NEWLINE?
       "{"
       NEWLINE?
       (
         MEMBER
         NEWLINE?
       )*
       "}"

4) Treat newlines as tokens. Whenever a non-newline token appears in the grammar and a newline token appears in the token stream, skip an arbitrary number of newline tokens to get to the next non-newline token. This is equivalent to replacing every non-newline terminal symbol X in the grammar with the (NEWLINE* X) sequence. You obviously wouldn't do that by hand. You've got a general purpose computer in front of you. Use it!
 The thing about most compiler construction tools is that they don't
 allow interaction between the context-guided tokenization, and they're
 not designed for the creation of backup-and-retry processing, or the
 insertion of virtual tokens into the token stream.

And don't use outdated crappy compiler construction tools. -- Rainer Deyke - rainerd eldwood.com
Sep 11 2009
prev sibling next sibling parent language_fan <foo bar.com.invalid> writes:
Mon, 14 Sep 2009 07:33:59 -0400, bearophile thusly wrote:

 But lot of people will judge D against more modern languages like C#,
 Scala or Java) and not against C.

Programmers often belong to three kinds of groups. First come the fans of traditionally weakly typed compiled languages (basic, c, c++). They have tried some "dynamic" or "academic" languages but did not like them. They fancy efficiency and close to metal feel. They think compilation to native code is the best way to produce programs, and think types should reflect the feature set of their cpu. They believe the syntax C uses was defined by their God. The second group started with interpreted languages built by amateurs (php, ruby, python, some game scripting language etc). They do not understand the meaning the types or compilation. They prefer writing short programs that usually seem to work. They hate formal specifications and proofs about program properties. They are usually writing simple web applications or some basic shareware utilies no one uses. They also hate trailing semicolons. The members of the last group have studied computer science and languages, in particular. They have found a pet academic language, typically a pure one, but paradigms may differ. In fact this is the group which uses something other than the hybrid object-oriented/procedural model. They appreciate a strong, orthogonal core language that scales cleanly. They are not scared of esoteric non-C-like syntax. They use languages that are not ready to take a step to the "real world" during the 70 next years. So yes, every group has a bit different expectations..
C++ isn't anymore complex than D2,<

I don't agree, see below.

It would help all of you if you could somehow formally specify how you measure language complexity. Is it the length of the grammar definition or something else? Otherwise these are just subjective opinions.
Sep 14 2009
prev sibling next sibling parent language_fan <foo bar.com.invalid> writes:
Mon, 14 Sep 2009 21:55:10 -0600, Rainer Deyke thusly wrote:

 language_fan wrote:
 The members of the last group have studied computer science and
 languages, in particular. They have found a pet academic language,
 typically a pure one, but paradigms may differ. In fact this is the
 group which uses something other than the hybrid
 object-oriented/procedural model. They appreciate a strong, orthogonal
 core language that scales cleanly. They are not scared of esoteric
 non-C-like syntax. They use languages that are not ready to take a step
 to the "real world" during the 70 next years.

Of the three types, this comes closest to describing me. Yet, I am completely self-taught, and my preferred language is still C++. (I wouldn't call it my pet language. I loathe C++, I just haven't found a suitable replacement yet.) Stereotypes are dangerous.

Indeed they are. My post should have been taken with a grain of salt. The idea was to show that languages in each group have their advantages and disadvantages. There is nothing wrong with being self-taught, many times people with formal education lack the passion many amateurs share. What is bad is that many people can only express their ideas in one kind of language, and that is usually their pet language. If you study Java, C#, C++, and D, they are all very similar to each other. Especially if you try to avoid learning all advanced features that are not common to all of them. In that case you don't know four different languages, but a single simple language mostly suitable for basic end user application development. On the other hand, knowing 40 academic languages will not get you far, either.
Sep 15 2009
prev sibling next sibling parent language_fan <foo bar.com.invalid> writes:
Tue, 15 Sep 2009 00:25:46 +0200, Lutger thusly wrote:

 That's a fancy way of saying that anyone who has not studied CS is a
 moron and therefore cannot understand what is good about languages, thus
 they lose any argument automatically. Am I right?

I just recommend learning basic concepts until terms like generational garbage collection, closure, register allocation, immutability, loop fusion, term rewriting, regular languages, type constructor, virtual constructor, and covariance do not scare you anymore. If something small like optional semicolons or some other syntactic nuance prevents you from finishing your job, how the heck are you supposed to build any real world programs? Just to put this to some perspective, syntax matters, but not much. Nowadays you can easily write a tool that parses stuff written in language X and outputs it in pretty printed form in language Y. This is what happens on .NET, for instance. Most of the languages there are just syntactic skins for the same common core language.
Sep 15 2009
prev sibling parent language_fan <foo bar.com.invalid> writes:
Tue, 15 Sep 2009 16:41:25 +0200, Lutger thusly wrote:

 language_fan wrote:
 
 Tue, 15 Sep 2009 00:25:46 +0200, Lutger thusly wrote:
 
 That's a fancy way of saying that anyone who has not studied CS is a
 moron and therefore cannot understand what is good about languages,
 thus they lose any argument automatically. Am I right?

I just recommend learning basic concepts until terms like generational garbage collection, closure, register allocation, immutability, loop fusion, term rewriting, regular languages, type constructor, virtual constructor, and covariance do not scare you anymore.

programmers make apps no-one uses using amateur tools | c-family users worship FOO and the rest are academics that use pure functional languages' part that tripped me up. You know, the majority of software isn't built by academics, NASA uses C mostly, etc. A little nuance wouldn't hurt here.

Ok. I did not even mean you should write all your programs in LISP or Prolog. The point is, once you know how to use various kinds of techniques, you can use whatever language you want. The problem is, most programmers only know 1-3 languages, and those languages are typically very similar to each other (e.g. Java, C, and C++). Some problems are inherently functional or easily expressed with regular expressions or as a logical constraint satisfaction problem. It does not make sense to write your own priority queue for each new task.
Sep 15 2009
prev sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Mon, 14 Sep 2009 18:53:51 -0400, Fawzi Mohamed <fmohamed mac.com> wrote:

 On 2009-09-14 17:07:00 +0200, "Robert Jacques" <sandford jhu.edu> said:

 On Mon, 14 Sep 2009 09:39:51 -0400, Leandro Lucarella  
 <llucax gmail.com>  wrote:
 Jeremie Pelletier, el 13 de septiembre a las 22:58 me escribiste:

 I understand your points for using a separate memory manager, and
 I agree with you that having less active allocations make for faster
 sweeps, no matter how little of them are scanned for pointers. However
 I just had an idea on how to implement generational collection on
 a non-moving GC which should solve your issues (and well, mines too)
 with the collector not being fast enough. I need to do some hacking on

objects/pages in each generation and modify that lists instead of moving objects. I can't remember the name of the paper so I can't find it now :S The problem with generational collectors (in D) is that you need read/write barriers to track inter-generational pointers (to be able to use pointers to younger generations in the older ones as roots when scanning), which can make the whole deal a little unpractical for a language that doesn't want to impose performance penalty to thing you wont use (I don't see a way to instrument read/writes to pointers to the GC only). This is why RC was always rejected as an algorithm for the GC in D, I think.
 my custom GC first, but I believe it could give yet another  
 performance
 boost. I'll add my memory manager to my list of code modules to make
 public :)


According to a blog post (http://www.sealiesoftware.com/blog/archive/2009/08/28/objc_explain_Thread-local_garb ge_collection.html) apparently this has allowed pause times similar to the pause times of the previous generational GC. (Except that the former is doing a full collect, and the later still has work to do) On that note, it would probably be a good idea if core.gc.BlkAttr supported shared and immutable state flags, which could be used to support a thread-local GC.

1) to allocate large objects that have a guard object it is a good idea to pass through the GC because if memory is tight a gc collection is triggered thereby possibly freeing some extra memory 2) using gc malloc is not faster than malloc, especially with several threads the single lock of the basic gc makes itself felt. for how I use D (not realtime) the two things I would like to see from new gc are: 1) multiple pools (at least one per cpu, with thread id hash to assign threads to a given pool). This to avoid the need of a global gc lock in the gc malloc, and if possible use memory close to the cpu when a thread is pinned, not to have really thread local memory, if you really need local memory different from the stack then maybe a separate process should be used. This is especially well doable with 64 bits, with 32 memory usage/fragmentation could become an issue. 2) multiple thread doing the collection (a main thread distributing the work to other threads (one per cpu), that do the mark phase using atomic ops). other better gc, less latency (but not at the cost of too much computation), would be nice to have, but are not a priority for my usage. Fawzi

For what it's worth, the whole point of thread-local GC is to do 1) and 2). For the purposes of clarity, thread-local GC refers to each thread having it's own GC for non-shared objects + a shared GC for shared objects. Each thread's GC may allocate and collect independently of each other (e.g. in parallel) without locking/atomics/etc.
Sep 14 2009