digitalmars.D - Re: Template Metaprogramming Made Easy (Huh?)

Nick Sabalausky Wrote:
 As someone who's been meaning to take a look at Scala, I'm very curious: 
 What did you dislike about it?


I'm somewhat reluctant to discuss Scala too much here as this is a D forum, but
since more than one person asked it's only fair to reply with something.  Now
this is just my opinion, subjective and ignorant as it may be so, please, its
horses for courses and YMMV.  Also I fully support bio-diversity in programming
languages and it's ridiculous to suggest that there is or should be only only
true language and then to can a language because its not your idea of the one
true language.  Regarding my writing style, I attempt to employ a little humour
(lame as it may be) so please take that into account before anyone decides to
get really angry at me.

Well here goes ...

1. Moved in with her for six months.  She's an overkill of a language,
seductive at first but then like a Black Widow spider eats her mate after
courtship.

http://en.wikipedia.org/wiki/Latrodectus_hesperus

2. Scala cannot make up her mind if she's a scripting language or serious
language.  Optional semicolons at the end of statements are really frustrating
for code style consistency.  Worst when sometimes you need them and you left
them out everywhere else for code style consistency.  (JavaScript has this
problem too and JS guru Douglas Crockford recommends semicolons always be used
in that language despite them being optional when statements clearly separated
by newlines.)

3. Half-baked embeddable XML support in the language looks like she borrowed
from ECMAScript's E4X.

4. Too many different ways of doing things.  All very interesting and no doubt
very clever but she needs to shave her hairy legs with Occam's razor before she
starts to look like the sister of Frankenstein's monster**.

http://en.wikipedia.org/wiki/Occam%27s_razor

4. Way too much of an academic approach to language design; appears to be a
grand experiment to see how many academic papers can be derived from it.

Google for "Scala experiment" with and without the quotes and you'll soon
realize she was designed in a lab from a kitchen sink full of PL body parts
like her brother in fiction**.

Read Cedric's blog June 2008 for example

http://beust.com/weblog/archives/000490.html

Still there's no such thing as a "failed experiment".  An experiment is just an
"experiment".  The light bulb wouldn't exist today if Edison wasn't as
tenacious as he was.  He simply treated "failed" experiments as just another
way of learning how not to make a light bulb.

5. Newcomers to the language will find it's type system concepts overwhelming -
co-variance and contra-variance etc.  (don't know how D2 will address this
better though). Yes these issues are important for OO libraries but feel there
must be a more practical way out of the language complexity.  Personally I
always kept away from the hairy and scary bits of C++; you don't need 'em in a
practical language.

I've heard Scala's argument that all the complexity is hidden in the libraries
so need to worry about it.  Unfortunately I don't believe her.  I learn a lot
about a language by studying the library code and expect it to be as easy to
read and understand as mainline code.

5. Not her fault (i.e. of the language), but after six months of courting Scala
with the Eclipse plugin, suffering IDE crash after crash and lost code I just
could not bring myself to suffering her any longer.

Read August 2009 comments by Tim at

http://blog.jayway.com/2009/03/12/scala-ide-support/

"Half a year after the above post, I�m still shocked at how badly the Scala
plug-in for Eclipse behaves. I�ve downloaded several variants on Eclipse over
the last half-year (currently 3.4.2), and NONE of them have been able to do
even basic things reliably with the Scala plug-in (or vise-versa)
...
without, say, basic knowledge of code (how to find a referenced method) and the
ability to run more than five minutes, I might as well go back to Vim. (Which I
think I�m going to have to do.)"

When I went looking for an Eclipse plugin for D a few weeks ago, I soon
discovered Descent.  It's not perfect by any means, but works well enough not
to be a thorn in your side and deserves credit for where it's at already.

6. In agreement with comments by Tony Arcieri (April 2009)

"I was initially excited about Scala but slowly grew discontented with it."

http://www.unlimitednovelty.com/2009/04/why-i-dont-like-scala.html

To sum up, after six months of living in, I felt I still wasn't getting
anywhere close to being intimate with her.  It really shouldn't take that long
to get up to speed with a new PL.

My advice for D2 language designers is not to copy every "cool" feature of
Scala.  For good balance of ideas, look at Rich Hickey's Clojure language.

http://clojure.org/

Sep 10 2009

Jeremie Pelletier <jeremiep gmail.com> writes:

Justin Johansson Wrote:
 5. Newcomers to the language will find it's type system concepts overwhelming
- co-variance and contra-variance etc.  (don't know how D2 will address this
better though). Yes these issues are important for OO libraries but feel there
must be a more practical way out of the language complexity.  Personally I
always kept away from the hairy and scary bits of C++; you don't need 'em in a
practical language.
 
 I've heard Scala's argument that all the complexity is hidden in the libraries
so need to worry about it.  Unfortunately I don't believe her.  I learn a lot
about a language by studying the library code and expect it to be as easy to
read and understand as mainline code.


I couldn't agree more, I learned how to use D by studying its runtime library
over the past few years. To me it is especially useful to study a runtime
library when it is used to implement features of the language, so you get a
clear grasp of what using these features imply. I lost count on how many neat
tricks I learned reading Andrei's metaprogramming code.

Sep 10 2009

language_fan <foo bar.com.invalid> writes:

Thu, 10 Sep 2009 20:25:04 -0400, Justin Johansson thusly wrote:

 2. Scala cannot make up her mind if she's a scripting language or
 serious language.  Optional semicolons at the end of statements are
 really frustrating for code style consistency.  Worst when sometimes you
 need them and you left them out everywhere else for code style
 consistency.  (JavaScript has this problem too and JS guru Douglas
 Crockford recommends semicolons always be used in that language despite
 them being optional when statements clearly separated by newlines.)


Ok, so you did not like optional semicolons.

 
 3. Half-baked embeddable XML support in the language looks like she
 borrowed from ECMAScript's E4X.


Ok, borrowing features is bad.

 
 4. Too many different ways of doing things.  All very interesting and no
 doubt very clever but she needs to shave her hairy legs with Occam's
 razor before she starts to look like the sister of Frankenstein's
 monster**.


Examples? I can tell you that D has too many looping constructs: goto, 
for, foreach, foreach_reverse, do-while, while, recursive functions, 
recursive templates.

 4. Way too much of an academic approach to language design; appears to
 be a grand experiment to see how many academic papers can be derived
 from it.


Ok, academic research is bad.

 5. Newcomers to the language will find it's type system concepts
 overwhelming - co-variance and contra-variance etc.  (don't know how D2
 will address this better though). Yes these issues are important for OO
 libraries but feel there must be a more practical way out of the
 language complexity.  Personally I always kept away from the hairy and
 scary bits of C++; you don't need 'em in a practical language.


I know many professional Java/C# coders who know the concepts of 
variances. The concepts are not new.. have you ever written code in Java 
(? extends Foo) etc. ? Java is a simple language, much simpler than D and 
even in Java land you need to care about these. I have also heard C++ 
developers pondering these issues. Giving them the proper names comes 
from the academia, though.

 I've heard Scala's argument that all the complexity is hidden in the
 libraries so need to worry about it.  Unfortunately I don't believe her.
  I learn a lot about a language by studying the library code and expect
 it to be as easy to read and understand as mainline code.


It is. The core language does not have e.g. arrays or AAs. Library code 
is known to be much more complex than ordinary application code.

I would not recommend studying a new language by reading the sources of 
libraries. It is like trying to learn quick-sort by reading the sources 
of JVM, ffmpeg or the Linux kernel. For instance there is the book 
'Programming in Scala', which is much better a starting point as the 
language also has some theoretical prerequisites. Reading the library 
code gets easier after reading the book.

 5. Not her fault (i.e. of the language), but after six months of
 courting Scala with the Eclipse plugin, suffering IDE crash after crash
 and lost code I just could not bring myself to suffering her any longer.


When Descent was new, it also sucked. It used to hang badly when it 
parsed partial code, like when one did not yet have a closing bracket for 
arrays. The Netbeans plugin for Scala is more stable.

Sep 11 2009

Justin Johansson <procode adam-dott-com.au> writes:

 Ok, so you did not like optional semicolons.



 3. Half-baked embeddable XML support in the language looks like she
 borrowed from ECMAScript's E4X.





 4. Too many different ways of doing things.  All very interesting and no


 for, foreach, foreach_reverse, do-while, while, recursive functions, 



 4. Way too much of an academic approach to language design; appears to





Further, I'd much rather see universities teach Scala in CS courses in
preference to any of C++, Java or D.  The latter are all something one can
learn on the job if they have a good founding in the fundamentals.  Since
Scala's designer studied under Nicholas Wirth, arguably Scala is the successor
of Modula-2 and Pascal (two languages developed by Wirth) and it wouldn't be
any surprise if Scala becomes the next teaching language.

My original point was that Scala being in my opinion too academic, was not for
my developer's taste; not that anything academic is bad.

 'Programming in Scala', which is much better a starting point as the 



 5. Not her fault (i.e. of the language), but after six months of
 courting Scala with the Eclipse plugin, suffering IDE crash after crash
 and lost code I just could not bring myself to suffering her any longer.


 When Descent was new, it also sucked. It used to hang badly when it 
 parsed partial code, like when one did not yet have a closing bracket for 
 arrays. The Netbeans plugin for Scala is more stable.



Anyway, thanks for your comments on my comments .. all in the spirit of good
debate.

Cheers
Justin

Sep 11 2009

bearophile <bearophileHUGS lycos.com> writes:

Justin Johansson:

I'm somewhat reluctant to discuss Scala too much here as this is a D forum,<

Discussing other languages is allowed here, especially if they show good things
or bad things that may help D development.

2. Scala cannot make up her mind if she's a scripting language or serious
language.<

Scala purpose is to try to be good for both small and large programs, that's
the meaning of Scala, that stands for "Scalable language". I'd like to see D2
become a little fitter for small programs.

Optional semicolons at the end of statements are really frustrating for code
style consistency.<

Semicolons are noise, they slow down programming a little. Better to design
future languages where newlines are enough. See also the Delight language,
that's better than normal D.

Read Cedric's blog June 2008 for example
http://beust.com/weblog/archives/000490.html

The comments to that blog post are more intelligent and useful than the main
post. See for example the comment by Amit Patel.

http://www.unlimitednovelty.com/2009/04/why-i-dont-like-scala.html<
Furthermore, Scala's object model is not only familiar, it's extremely
well-studied and well-optimized. The JVM provides immense capability to inline
method calls, which means calls which span multiple objects can be condensed
down to a single function call. This is because the Smalltalk-inspired illusion
that these objects are receiving and sending messages is completely suspended,
and objects are treated in C++-style as mere chunks of state, thus an inlined
method call can act on many of them at once as if they were simple chunks of
state. In Reia, all objects are concurrent, share no state, and can only
communicate with messages. Inlining calls across objects is thoroughly
impossible since sending messages in Reia is not some theoretical construct,
it's what really happens and cannot simply be abstracted away into a function
call which mutates the state of multiple objects.<

D compilers are generally not even able to inline most virtual calls, so Reia
doesn't look like a fitting design for a language that must be fast as D. Reia
design looks good for a higher level language. Scala is designed to be faster
than Reia. Maybe someday people will find ways to efficiently compile Reia too,
similar things have appened several times.

http://clojure.org/<

Clojure is not OOP, and it lacks several of the modern functional features
(like a powerful type system). And it's slow. I don't like it a lot, despite
its management of immutability is cute.
Something about this topic:
http://blog.higher-order.net/2009/02/01/understanding-clojures-persistentvector-implementation/
http://blog.higher-order.net/2009/09/08/understanding-clojures-persistenthashmap-deftwice/

Bye,
bearophile

Sep 11 2009

Justin Johansson <procode adam-dott-com.au> writes:

bearophile Wrote:
 Justin Johansson:
I'm somewhat reluctant to discuss Scala too much here as this is a D forum,<




Okay.  Thanks for mentioning that, bearophile, and your other reply comments
noted.

Since I'm still feeling D around the edges, would you mind saying what salient
things there are about D that presumably attracts to the language.  It just
helps to know why others are here as one ventures into new territory.

Cheers
Justin Johansson

Sep 11 2009

bearophile <bearophileHUGS lycos.com> writes:

Justin Johansson:

would you mind saying what salient things there are about D that presumably
attracts to the language.  It just helps to know why others are here as one
ventures into new territory.<


That's not an easy question. This is a personal answer, other people will like
other sides of D. I like D1/D2 for:
- I don't think of it as a propetary language, like C#.
- Sometimes I want the freedom to use memory as I like, with structs, and even
unions. If you have large datasets you find that using more than 20 bytes for a
number as in Python doesn't help. Values also reduce indirection, this speeds
up things. This allows a more efficient usage of memory, and this helps
increase cache locality, that increases performance. Unfortunately GC-managed D
pointers can't be tagged, so I have to use memory from the C heap for them. And
sometimes you need pointers. That's why I'd like D to have more/better ways to
increase safety when using pointers (like using memory regions when not in
release mode, etc).
- I like this newsgroups, I can talk to people, and they sometimes answer my
numerous questions. I am learning a lot. Sometimes I receive no answers, but
it's acceptable. For its nature this newsgroup attracts some strange people too.
- I often use Python, it's the language I like more, but for some purposes it's
too much slow. And I am tired of writing vectorized code in NumPy and the like.
Cython reference count makes me sick and ShedSkin while nice is a failed
experiment. D feels like freedom, while sometimes using Python feels like
programming with mittens for me.
- There are some things that I'd like to see in a language, and D being in
development still and being not controlled by an ivory tower committee give me
the illusion to see some of my ideas realized. So far I haven't influenced a
lot the development of D. On the other hand if everyone can influence a lot the
language the result may be a patchwork. So some dynamic compromise has to be
found every day.
- D looks a lot like C, yet in D I can write code several times faster than C.
Sometimes 5-10 times faster. This is an enormous difference.
- I am quite sensitive to syntax noise, boilerplate code. I like elegance,
clarity in semantics, I hate corner cases, yet I want a language that's
efficient, readable, and the less bug-prone as possible. C++ looks too much
complex for me. D1 is simple enough for me, and D2 is getting a bit too much
complex. I may appreciate the idea of a D 1.5 language that fixes some holes
and limits of D1 while keeping language simple enough (struct constructors, and
few other things. Such things don't significantly increase the difficulty in
using the language).
- I like how D doesn't totally ignore safety as C does, in D sometimes the
default way is the safer one, and the unsafe way is used only where you ask for
it.  I'd like to see more safeties added to D, like optional run-time and
compile-time integral overflow tests, some pointer safety, better template
error messages (template constraints help some in such regard), stack traces,
less compiler bugs, safer casts (in C# you need explicit casts to convert
double => float), a safer printf, some optional annotations inspired by Plint
(a lint program) to give more semantics to the compiler, that can be used to
both speed up code and avoid bugs. There's lot that can be done in this regard.
And release-mode performance can be usually kept unchanged.
- I like how templates allow me to do some things that are doable only in less
common languages like Lisp and Haskell, or in dynamic languages.
- To use D I don't need to fill forms, receive emails, pay, install with an
installer, or use an IDE. The compiler is free, you just need to download a
zip, and recently such zip is well organized too inside. I uncompress the zip,
seth a path or two and I am already able to use it.
- The LDC compiler on Linux produces binaries that are efficient almost as C++
and sometimes more. Someday LDC will be available on Windows too. LDC
developers are good people, they fix things very quickly (often in 24 hours),
and they don't ignore user requests. For example in LDC the == among
associative arrays now works. LDC developers are almost as serious people as
LLVM devs, but no one gets paid for LDC (while the head of LLVM is paid by
Apple).
- D contains many tricks and handy features that save lot of time and make
programming shorter and simpler. Some of such features are half-unfinished
(some GC/GC-pointers semantics, module system, unittest system, contract
programming, and several other things) and Walter doesn't seem willing to
finish them soon, but having them partially unfinished is better than not
having them.

So in summary I like the freedom D gives me in using memory, and the freedom to
program in the style I see most fit for a program, its work-in-progress nature
that gives the illusion to be able to influence it in good ways, its simple
enough nature, its handy features, its clear and short enough syntax (even if
there are ways to improve it still). I like people in the D community, because
they sometimes understand what's the way to improve the language.

But probably the top two reasons for me are that it allows me to write fast
programs and at the same time it's not a very bug-prone language. Helping me
avoid bugs saves a ton of my programming time. Saves my time both when I
program and when I run the program. Compared to this all other things are
secondary.

Bye,
bearophile

Sep 11 2009

bearophile <bearophileHUGS lycos.com> writes:

Saves my time both when I program and when I run the program. Compared to this
all other things are secondary.<


"Saving time" is one of the most important qualities of a programming language.
A language can save time in many ways:
- If it's easy to find with Google, if it's fast to download and install. If
you have to pay for it you need waste some time. Sometimes installers help save
time, but a zip file too can sometimes save time compared to an ugly installer.
If you don't have to fill forms to download it it saves your time. Having an
integrated editor, or having a IDE+compiler can save programming time. A good
IDE can even turn a boring language like Java in an usable and quite useful
one. Languages like C# are designed to almost require an IDE.
- Having good online documentation and a good help can save lot of time.
- A rich, easy to use and well debugged standard library can save a lot of
time. Having a community of code (like Python modules you can find online to do
almost everything, that are written in an uniform style that's easy to read and
understand) can save a very large amount of time, even months.
- If it's simple or similar to other languages, if its semantics is clear, this
saves some time to learn it, sometimes many months.
- A compact syntax saves a little programming time. A clear syntax improves
readability, and this saves a lot of time when the program has to be debugged,
modified, improved or just undertood.
- A clear and high level semantics allows the programmer to think less about
details, and this speeds up the invention or implementation of algorithms, and
saves time (Python is among the best at this)..
- If it helps avoid bugs or remove them it can save lot of programming time.
- Some built-in features of the language can save some programming time.
- If the compilation process is simple this saves time (see the Bud tool). If
the language is designed to be compiled quickly (or to not require compilation)
this saves time.
- If programs run quickly it saves some time.

A good language has to try to save time in all those ways and more.

Bye,
bearophile

Sep 11 2009

Justin Johansson <procode adam-dott-com.au> writes:

 - If programs run quickly it saves some time.
 
 A good language has to try to save time in all those ways and more.


Tks bearophile for that extensive writeup.  A good read.

btw. Downloaded the Bud tool (on linux) but couldn't get it to compile.  First
had to rename usage of "macro" to "makro" then it bitched about some missing
module so I gave up.

-- Justin

Sep 13 2009

Jeremie Pelletier <jeremiep gmail.com> writes:

bearophile Wrote:

 Justin Johansson:
 
would you mind saying what salient things there are about D that presumably
attracts to the language.  It just helps to know why others are here as one
ventures into new territory.<


 That's not an easy question. This is a personal answer, other people will like
other sides of D. I like D1/D2 for:
 - I don't think of it as a propetary language, like C#.
 - Sometimes I want the freedom to use memory as I like, with structs, and even
unions. If you have large datasets you find that using more than 20 bytes for a
number as in Python doesn't help. Values also reduce indirection, this speeds
up things. This allows a more efficient usage of memory, and this helps
increase cache locality, that increases performance. Unfortunately GC-managed D
pointers can't be tagged, so I have to use memory from the C heap for them. And
sometimes you need pointers. That's why I'd like D to have more/better ways to
increase safety when using pointers (like using memory regions when not in
release mode, etc).


I haven't had to use the C heap whatsoever so far in D, could you give me an
example of where you need it? In fact, the *only* place I use the C heap is in
my garbage collector's internals, for pool structs and mark ranges. I use
pointers to GC memory all the time too, there are plenty of algorithms,
especially in loops, that can run faster with pointer arithmetic than slices
and it's still the fastest way to pass struct references around.

 - I like this newsgroups, I can talk to people, and they sometimes answer my
numerous questions. I am learning a lot. Sometimes I receive no answers, but
it's acceptable. For its nature this newsgroup attracts some strange people too.
 - I often use Python, it's the language I like more, but for some purposes
it's too much slow. And I am tired of writing vectorized code in NumPy and the
like. Cython reference count makes me sick and ShedSkin while nice is a failed
experiment. D feels like freedom, while sometimes using Python feels like
programming with mittens for me.
 - There are some things that I'd like to see in a language, and D being in
development still and being not controlled by an ivory tower committee give me
the illusion to see some of my ideas realized. So far I haven't influenced a
lot the development of D. On the other hand if everyone can influence a lot the
language the result may be a patchwork. So some dynamic compromise has to be
found every day.


I also like this community driven model, but this forum has more people
submitting ideas than people able to implement them on time, I'm pretty sure
the TODO list is rather huge at this time :) I for one much prefer D
development the way it is now than the "working group" model used by the W3C or
Khronos for example.

The public bugzilla is really nice too, once you get used to it, one of the
issues I submitted got fixed in 2.032, I've also sent maybe 3 or 4 patches to
the compiler source in other issues so far too, hopefully they'll be used in
2.033!

 - D looks a lot like C, yet in D I can write code several times faster than C.
Sometimes 5-10 times faster. This is an enormous difference.


Indeed, and sometimes it's way faster than that. I've ported many C headers to
D and I'm always amazed at how many things I can throw out, just the DirectX
headers were at least 50% smaller in D and MUCH easier to read. Such simplicity
is also reflected in the compiler by having quite a lot less tokens and parse
nodes to create and analyze.

I must admit however that I sometimes miss the C preprocessor, or at least wish
mixins had a syntax closer to that used by the C preprocessor. But it's a good
idea to keep D without a preprocessor, its much better for everything to have a
scope.

 - I am quite sensitive to syntax noise, boilerplate code. I like elegance,
clarity in semantics, I hate corner cases, yet I want a language that's
efficient, readable, and the less bug-prone as possible. C++ looks too much
complex for me. D1 is simple enough for me, and D2 is getting a bit too much
complex. I may appreciate the idea of a D 1.5 language that fixes some holes
and limits of D1 while keeping language simple enough (struct constructors, and
few other things. Such things don't significantly increase the difficulty in
using the language).


C++ isn't anymore complex than D2, its syntax just isn't as elegant. Other than
multiple inheritance which is partially solved through object composition, I
can't think of many features C++ has over D2. I can name quite a few features
D2 has over C++ :)

What I like about D is that while its elegant, it still allows for dirty work
to be done when you need it. You don't need to code your application core in C
and your application behavior in a scripting language on top of the C core. D
allows you to write it all in one language with the same productivity, if not
better productivity for not having to write the abstraction layer between C and
scripting.

 - I like how D doesn't totally ignore safety as C does, in D sometimes the
default way is the safer one, and the unsafe way is used only where you ask for
it.  I'd like to see more safeties added to D, like optional run-time and
compile-time integral overflow tests, some pointer safety, better template
error messages (template constraints help some in such regard), stack traces,
less compiler bugs, safer casts (in C# you need explicit casts to convert
double => float), a safer printf, some optional annotations inspired by Plint
(a lint program) to give more semantics to the compiler, that can be used to
both speed up code and avoid bugs. There's lot that can be done in this regard.
And release-mode performance can be usually kept unchanged.


Stack traces is a feature for the runtime, I made one for mine, which shows a
dialog window with the stack trace, current registers values, loaded modules
and whatnot. Took me almost a week to piece together my CodeView reader, I
still need a DWARF reader for linux. I'm gonna try and implement it in druntime
and submit the patch to bugzilla.

 - I like how templates allow me to do some things that are doable only in less
common languages like Lisp and Haskell, or in dynamic languages.
 - To use D I don't need to fill forms, receive emails, pay, install with an
installer, or use an IDE. The compiler is free, you just need to download a
zip, and recently such zip is well organized too inside. I uncompress the zip,
seth a path or two and I am already able to use it.
 - The LDC compiler on Linux produces binaries that are efficient almost as C++
and sometimes more. Someday LDC will be available on Windows too. LDC
developers are good people, they fix things very quickly (often in 24 hours),
and they don't ignore user requests. For example in LDC the == among
associative arrays now works. LDC developers are almost as serious people as
LLVM devs, but no one gets paid for LDC (while the head of LLVM is paid by
Apple).
 - D contains many tricks and handy features that save lot of time and make
programming shorter and simpler. Some of such features are half-unfinished
(some GC/GC-pointers semantics, module system, unittest system, contract
programming, and several other things) and Walter doesn't seem willing to
finish them soon, but having them partially unfinished is better than not
having them.


There are hundreds of things on Walter's list, he can't magically do them all
overnight :)

 
 So in summary I like the freedom D gives me in using memory, and the freedom
to program in the style I see most fit for a program, its work-in-progress
nature that gives the illusion to be able to influence it in good ways, its
simple enough nature, its handy features, its clear and short enough syntax
(even if there are ways to improve it still). I like people in the D community,
because they sometimes understand what's the way to improve the language.
 
 But probably the top two reasons for me are that it allows me to write fast
programs and at the same time it's not a very bug-prone language. Helping me
avoid bugs saves a ton of my programming time. Saves my time both when I
program and when I run the program. Compared to this all other things are
secondary.
 
 Bye,
 bearophile

Sep 13 2009

Tom S <h3r3tic remove.mat.uni.torun.pl> writes:

Jeremie Pelletier wrote:
 I haven't had to use the C heap whatsoever so far in D, could you give me an
example of where you need it? In fact, the *only* place I use the C heap is in
my garbage collector's internals, for pool structs and mark ranges. I use
pointers to GC memory all the time too, there are plenty of algorithms,
especially in loops, that can run faster with pointer arithmetic than slices
and it's still the fastest way to pass struct references around.


I use the C heap a lot when I need slabs of memory that the GC should 
not look into for performance reasons. This includes images/textures, 
mesh data and some data structures that I release manually - again, for 
efficiency reasons.

 - I like how D doesn't totally ignore safety as C does, in D sometimes the
default way is the safer one, and the unsafe way is used only where you ask for
it.  I'd like to see more safeties added to D, like optional run-time and
compile-time integral overflow tests, some pointer safety, better template
error messages (template constraints help some in such regard), stack traces,
less compiler bugs, safer casts (in C# you need explicit casts to convert
double => float), a safer printf, some optional annotations inspired by Plint
(a lint program) to give more semantics to the compiler, that can be used to
both speed up code and avoid bugs. There's lot that can be done in this regard.
And release-mode performance can be usually kept unchanged.


 Stack traces is a feature for the runtime, I made one for mine, which shows a
dialog window with the stack trace, current registers values, loaded modules
and whatnot. Took me almost a week to piece together my CodeView reader, I
still need a DWARF reader for linux. I'm gonna try and implement it in druntime
and submit the patch to bugzilla.


Tango's runtime already does stack tracing on Windows and *NIX, however 
its CV parser is subject to some licensing issues :( Perhaps you could 
release yours under some liberal license so it can be plugged there? :)


-- 
Tomasz Stachowiak
http://h3.team0xf.com/
h3/h3r3tic on #D freenode

Sep 13 2009

Jeremie Pelletier <jeremiep gmail.com> writes:

Tom S Wrote:

 Jeremie Pelletier wrote:
 I haven't had to use the C heap whatsoever so far in D, could you give me an
example of where you need it? In fact, the *only* place I use the C heap is in
my garbage collector's internals, for pool structs and mark ranges. I use
pointers to GC memory all the time too, there are plenty of algorithms,
especially in loops, that can run faster with pointer arithmetic than slices
and it's still the fastest way to pass struct references around.


 I use the C heap a lot when I need slabs of memory that the GC should 
 not look into for performance reasons. This includes images/textures, 
 mesh data and some data structures that I release manually - again, for 
 efficiency reasons.


The garbage collector in D already mark allocations which contains pointers and
scans these only. If you want to know if a type contains pointers, check the
'flags' property of the typeinfo or classinfo, test for bit0 and bit1
respectively. This is what the GC uses at runtime when allocating memory to
know if it should tag the allocation as containing possible pointers.

I myself allocate all my meshes and textures directly on the GC and I'm pretty
sure its faster than C's malloc and much safer.

 - I like how D doesn't totally ignore safety as C does, in D sometimes the
default way is the safer one, and the unsafe way is used only where you ask for
it.  I'd like to see more safeties added to D, like optional run-time and
compile-time integral overflow tests, some pointer safety, better template
error messages (template constraints help some in such regard), stack traces,
less compiler bugs, safer casts (in C# you need explicit casts to convert
double => float), a safer printf, some optional annotations inspired by Plint
(a lint program) to give more semantics to the compiler, that can be used to
both speed up code and avoid bugs. There's lot that can be done in this regard.
And release-mode performance can be usually kept unchanged.


 Stack traces is a feature for the runtime, I made one for mine, which shows a
dialog window with the stack trace, current registers values, loaded modules
and whatnot. Took me almost a week to piece together my CodeView reader, I
still need a DWARF reader for linux. I'm gonna try and implement it in druntime
and submit the patch to bugzilla.


 Tango's runtime already does stack tracing on Windows and *NIX, however 
 its CV parser is subject to some licensing issues :( Perhaps you could 
 release yours under some liberal license so it can be plugged there? :)
 


Sure, I wouldn't mind at all, I'm not into licenses myself so I might just
release it to public domain. I'll try and get a standalone package ready and
post it somewhere, I just don't know where yet :x

Sep 13 2009

Tom S <h3r3tic remove.mat.uni.torun.pl> writes:

Jeremie Pelletier wrote:
 Tom S Wrote:
 
 Jeremie Pelletier wrote:
 I haven't had to use the C heap whatsoever so far in D, could you give me an
example of where you need it? In fact, the *only* place I use the C heap is in
my garbage collector's internals, for pool structs and mark ranges. I use
pointers to GC memory all the time too, there are plenty of algorithms,
especially in loops, that can run faster with pointer arithmetic than slices
and it's still the fastest way to pass struct references around.


 not look into for performance reasons. This includes images/textures, 
 mesh data and some data structures that I release manually - again, for 
 efficiency reasons.


 The garbage collector in D already mark allocations which contains pointers
and scans these only. If you want to know if a type contains pointers, check
the 'flags' property of the typeinfo or classinfo, test for bit0 and bit1
respectively. This is what the GC uses at runtime when allocating memory to
know if it should tag the allocation as containing possible pointers.


Yea I know, but I want structures with pointers manually managed as well.


 I myself allocate all my meshes and textures directly on the GC and I'm pretty
sure its faster than C's malloc and much safer.


Hm, why would it be faster with the GC than malloc? I'm pretty sure it's 
the opposite :P Plus, I could use a specialized malloc implementation, 
like TLSF.

 
 - I like how D doesn't totally ignore safety as C does, in D sometimes the
default way is the safer one, and the unsafe way is used only where you ask for
it.  I'd like to see more safeties added to D, like optional run-time and
compile-time integral overflow tests, some pointer safety, better template
error messages (template constraints help some in such regard), stack traces,
less compiler bugs, safer casts (in C# you need explicit casts to convert
double => float), a safer printf, some optional annotations inspired by Plint
(a lint program) to give more semantics to the compiler, that can be used to
both speed up code and avoid bugs. There's lot that can be done in this regard.
And release-mode performance can be usually kept unchanged.




 its CV parser is subject to some licensing issues :( Perhaps you could 
 release yours under some liberal license so it can be plugged there? :)


 Sure, I wouldn't mind at all, I'm not into licenses myself so I might just
release it to public domain. I'll try and get a standalone package ready and
post it somewhere, I just don't know where yet :x


Sweet :D As for a place, there are plenty of options, e.g. 
http://dsource.org/projects/scrapple/ or a separate dsource project.



-- 
Tomasz Stachowiak
http://h3.team0xf.com/
h3/h3r3tic on #D freenode

Sep 13 2009

Jeremie Pelletier <jeremiep gmail.com> writes:

Tom S Wrote:

 Jeremie Pelletier wrote:
 Tom S Wrote:
 
 Jeremie Pelletier wrote:
 I haven't had to use the C heap whatsoever so far in D, could you give me an
example of where you need it? In fact, the *only* place I use the C heap is in
my garbage collector's internals, for pool structs and mark ranges. I use
pointers to GC memory all the time too, there are plenty of algorithms,
especially in loops, that can run faster with pointer arithmetic than slices
and it's still the fastest way to pass struct references around.


 not look into for performance reasons. This includes images/textures, 
 mesh data and some data structures that I release manually - again, for 
 efficiency reasons.


 The garbage collector in D already mark allocations which contains pointers
and scans these only. If you want to know if a type contains pointers, check
the 'flags' property of the typeinfo or classinfo, test for bit0 and bit1
respectively. This is what the GC uses at runtime when allocating memory to
know if it should tag the allocation as containing possible pointers.


 Yea I know, but I want structures with pointers manually managed as well.


Then just inform the GC to not scan the allocations you want, or better yet
have a static ctor modify the flag of the typeinfo you don't want scanned.

 I myself allocate all my meshes and textures directly on the GC and I'm pretty
sure its faster than C's malloc and much safer.


 Hm, why would it be faster with the GC than malloc? I'm pretty sure it's 
 the opposite :P Plus, I could use a specialized malloc implementation, 
 like TLSF.


The D GC is already specialized, and given its being used quite a lot in D,
there are good chances its already sitting in the CPU cache, its heap already
having the available memory block waiting on a freelist, or if the alloc is
more than 0x1000 bytes, the pages available in a pool. You'd need to use malloc
quite a lot to get the same optimal performance, and mixing the two would
affect the performance of both.

 - I like how D doesn't totally ignore safety as C does, in D sometimes the
default way is the safer one, and the unsafe way is used only where you ask for
it.  I'd like to see more safeties added to D, like optional run-time and
compile-time integral overflow tests, some pointer safety, better template
error messages (template constraints help some in such regard), stack traces,
less compiler bugs, safer casts (in C# you need explicit casts to convert
double => float), a safer printf, some optional annotations inspired by Plint
(a lint program) to give more semantics to the compiler, that can be used to
both speed up code and avoid bugs. There's lot that can be done in this regard.
And release-mode performance can be usually kept unchanged.




 its CV parser is subject to some licensing issues :( Perhaps you could 
 release yours under some liberal license so it can be plugged there? :)


 Sure, I wouldn't mind at all, I'm not into licenses myself so I might just
release it to public domain. I'll try and get a standalone package ready and
post it somewhere, I just don't know where yet :x


 Sweet :D As for a place, there are plenty of options, e.g. 
 http://dsource.org/projects/scrapple/ or a separate dsource project.


I thought of that, but I don't feel like opening a project for just a few
random code snippets or standalone classes. I think I'll just post it in this
forum and let interested people grab it for now.

Sep 13 2009

Tom S <h3r3tic remove.mat.uni.torun.pl> writes:

Jeremie Pelletier wrote:
 Tom S Wrote:
 
 Jeremie Pelletier wrote:
 I myself allocate all my meshes and textures directly on the GC and I'm pretty
sure its faster than C's malloc and much safer.


 the opposite :P Plus, I could use a specialized malloc implementation, 
 like TLSF.


 The D GC is already specialized, and given its being used quite a lot in D,
there are good chances its already sitting in the CPU cache, its heap already
having the available memory block waiting on a freelist, or if the alloc is
more than 0x1000 bytes, the pages available in a pool. You'd need to use malloc
quite a lot to get the same optimal performance, and mixing the two would
affect the performance of both.


It might be specialized for _something_, but it definitely isn't 
real-time systems. I'd say with my use cases there's a very poor chance 
the GC is sitting in the CPU cache since most of the time my memory is 
preallocated and managed by specialized structures and/or malloc. I've 
found that using the GC only for the hard-to-manually-manage objects 
works best. The rest is handled by malloc and the GC has a very shallow 
vision of the world thus its collection runs are very fast. Of course 
there's a drawback that both the GC and malloc will have some pages 
cached, wasting memory, but I don't let the GC touch too much so it 
should be minimal. YMMV of course - all depends on the memory allocation 
patterns of the application.


 - I like how D doesn't totally ignore safety as C does, in D sometimes the
default way is the safer one, and the unsafe way is used only where you ask for
it.  I'd like to see more safeties added to D, like optional run-time and
compile-time integral overflow tests, some pointer safety, better template
error messages (template constraints help some in such regard), stack traces,
less compiler bugs, safer casts (in C# you need explicit casts to convert
double => float), a safer printf, some optional annotations inspired by Plint
(a lint program) to give more semantics to the compiler, that can be used to
both speed up code and avoid bugs. There's lot that can be done in this regard.
And release-mode performance can be usually kept unchanged.




 its CV parser is subject to some licensing issues :( Perhaps you could 
 release yours under some liberal license so it can be plugged there? :)




 http://dsource.org/projects/scrapple/ or a separate dsource project.


 I thought of that, but I don't feel like opening a project for just a few
random code snippets or standalone classes. I think I'll just post it in this
forum and let interested people grab it for now.


WORKSFORME :)


-- 
Tomasz Stachowiak
http://h3.team0xf.com/
h3/h3r3tic on #D freenode

Sep 13 2009

Jeremie Pelletier <jeremiep gmail.com> writes:

Tom S Wrote:

 Jeremie Pelletier wrote:
 Tom S Wrote:
 
 Jeremie Pelletier wrote:
 I myself allocate all my meshes and textures directly on the GC and I'm pretty
sure its faster than C's malloc and much safer.


 the opposite :P Plus, I could use a specialized malloc implementation, 
 like TLSF.


 The D GC is already specialized, and given its being used quite a lot in D,
there are good chances its already sitting in the CPU cache, its heap already
having the available memory block waiting on a freelist, or if the alloc is
more than 0x1000 bytes, the pages available in a pool. You'd need to use malloc
quite a lot to get the same optimal performance, and mixing the two would
affect the performance of both.


 It might be specialized for _something_, but it definitely isn't 
 real-time systems. I'd say with my use cases there's a very poor chance 
 the GC is sitting in the CPU cache since most of the time my memory is 
 preallocated and managed by specialized structures and/or malloc. I've 
 found that using the GC only for the hard-to-manually-manage objects 
 works best. The rest is handled by malloc and the GC has a very shallow 
 vision of the world thus its collection runs are very fast. Of course 
 there's a drawback that both the GC and malloc will have some pages 
 cached, wasting memory, but I don't let the GC touch too much so it 
 should be minimal. YMMV of course - all depends on the memory allocation 
 patterns of the application.


I understand your points for using a separate memory manager, and I agree with
you that having less active allocations make for faster sweeps, no matter how
little of them are scanned for pointers. However I just had an idea on how to
implement generational collection on a non-moving GC which should solve your
issues (and well, mines too) with the collector not being fast enough. I need
to do some hacking on my custom GC first, but I believe it could give yet
another performance boost. I'll add my memory manager to my list of code
modules to make public :)

Sep 13 2009

Tom S <h3r3tic remove.mat.uni.torun.pl> writes:

Jeremie Pelletier wrote:
 Tom S Wrote:
 
 Jeremie Pelletier wrote:
 Tom S Wrote:

 Jeremie Pelletier wrote:
 I myself allocate all my meshes and textures directly on the GC and I'm pretty
sure its faster than C's malloc and much safer.


 the opposite :P Plus, I could use a specialized malloc implementation, 
 like TLSF.




 real-time systems. I'd say with my use cases there's a very poor chance 
 the GC is sitting in the CPU cache since most of the time my memory is 
 preallocated and managed by specialized structures and/or malloc. I've 
 found that using the GC only for the hard-to-manually-manage objects 
 works best. The rest is handled by malloc and the GC has a very shallow 
 vision of the world thus its collection runs are very fast. Of course 
 there's a drawback that both the GC and malloc will have some pages 
 cached, wasting memory, but I don't let the GC touch too much so it 
 should be minimal. YMMV of course - all depends on the memory allocation 
 patterns of the application.


 I understand your points for using a separate memory manager, and I agree with
you that having less active allocations make for faster sweeps, no matter how
little of them are scanned for pointers. However I just had an idea on how to
implement generational collection on a non-moving GC which should solve your
issues (and well, mines too) with the collector not being fast enough. I need
to do some hacking on my custom GC first, but I believe it could give yet
another performance boost. I'll add my memory manager to my list of code
modules to make public :)


Sounds great, I can't wait! :D


-- 
Tomasz Stachowiak
http://h3.team0xf.com/
h3/h3r3tic on #D freenode

Sep 13 2009

Nick B <nickB gmail.com> writes:

Jeremie Pelletier wrote:
 Tom S Wrote:
 
 Jeremie Pelletier wrote:
 Tom S Wrote:

 Jeremie Pelletier wrote:
 I myself allocate all my meshes and textures directly on the GC and I'm pretty
sure its faster than C's malloc and much safer.


 the opposite :P Plus, I could use a specialized malloc implementation, 
 like TLSF.




 real-time systems. I'd say with my use cases there's a very poor chance 
 the GC is sitting in the CPU cache since most of the time my memory is 
 preallocated and managed by specialized structures and/or malloc. I've 
 found that using the GC only for the hard-to-manually-manage objects 
 works best. The rest is handled by malloc and the GC has a very shallow 
 vision of the world thus its collection runs are very fast. Of course 
 there's a drawback that both the GC and malloc will have some pages 
 cached, wasting memory, but I don't let the GC touch too much so it 
 should be minimal. YMMV of course - all depends on the memory allocation 
 patterns of the application.


 I understand your points for using a separate memory manager, and I agree with
you that having less active allocations make for faster sweeps, no matter how
little of them are scanned for pointers. However I just had an idea on how to
implement generational collection on a non-moving GC which should solve your
issues (and well, mines too) with the collector not being fast enough. I need
to do some hacking on my custom GC first, but I believe it could give yet
another performance boost. I'll add my memory manager to my list of code
modules to make public :)



If the code is really usefull, why not offer it to the Tango team, for 
formal inclusion  in the next release ?

Nick B

Sep 13 2009

Jeremie Pelletier <jeremiep gmail.com> writes:

Nick B Wrote:

 Jeremie Pelletier wrote:
 Tom S Wrote:
 
 Jeremie Pelletier wrote:
 Tom S Wrote:

 Jeremie Pelletier wrote:
 I myself allocate all my meshes and textures directly on the GC and I'm pretty
sure its faster than C's malloc and much safer.


 the opposite :P Plus, I could use a specialized malloc implementation, 
 like TLSF.




 real-time systems. I'd say with my use cases there's a very poor chance 
 the GC is sitting in the CPU cache since most of the time my memory is 
 preallocated and managed by specialized structures and/or malloc. I've 
 found that using the GC only for the hard-to-manually-manage objects 
 works best. The rest is handled by malloc and the GC has a very shallow 
 vision of the world thus its collection runs are very fast. Of course 
 there's a drawback that both the GC and malloc will have some pages 
 cached, wasting memory, but I don't let the GC touch too much so it 
 should be minimal. YMMV of course - all depends on the memory allocation 
 patterns of the application.


 I understand your points for using a separate memory manager, and I agree with
you that having less active allocations make for faster sweeps, no matter how
little of them are scanned for pointers. However I just had an idea on how to
implement generational collection on a non-moving GC which should solve your
issues (and well, mines too) with the collector not being fast enough. I need
to do some hacking on my custom GC first, but I believe it could give yet
another performance boost. I'll add my memory manager to my list of code
modules to make public :)


 
 If the code is really usefull, why not offer it to the Tango team, for 
 formal inclusion  in the next release ?
 
 Nick B


Because I dropped support for D1 long ago. If either the Tango or Phobos team
like my code once I publish it, they are free to adapt it for their runtime. 

I rewrote the GC from scratch and optimized over the past 2 years to support my
custom D runtime. It cannot be used as-is with neither phobos or tango without
either changing the public interface of the GC or rewriting every runtime
routine calling into the GC. I would only release it to public domain as an
example of how to implement a tracing generational non-moving GC. I still need
to implement the generational part, but I got the general algorithm down on
paper today so I should have it working sometime this week.

I'm not a big fan of code licenses and therefore like to write most of my code
myself, if only to learn how it works. I rarely mind people asking for my code
either, so long as I get credited for it :)

Sep 14 2009

Leandro Lucarella <llucax gmail.com> writes:

Jeremie Pelletier, el 13 de septiembre a las 22:58 me escribiste:
 Tom S Wrote:
 
 Jeremie Pelletier wrote:
 Tom S Wrote:
 
 Jeremie Pelletier wrote:
 I myself allocate all my meshes and textures directly on the GC and I'm pretty
sure its faster than C's malloc and much safer.


 the opposite :P Plus, I could use a specialized malloc implementation, 
 like TLSF.


 The D GC is already specialized, and given its being used quite a lot in D,
there are good chances its already sitting in the CPU cache, its heap already
having the available memory block waiting on a freelist, or if the alloc is
more than 0x1000 bytes, the pages available in a pool. You'd need to use malloc
quite a lot to get the same optimal performance, and mixing the two would
affect the performance of both.


 It might be specialized for _something_, but it definitely isn't 
 real-time systems. I'd say with my use cases there's a very poor chance 
 the GC is sitting in the CPU cache since most of the time my memory is 
 preallocated and managed by specialized structures and/or malloc. I've 
 found that using the GC only for the hard-to-manually-manage objects 
 works best. The rest is handled by malloc and the GC has a very shallow 
 vision of the world thus its collection runs are very fast. Of course 
 there's a drawback that both the GC and malloc will have some pages 
 cached, wasting memory, but I don't let the GC touch too much so it 
 should be minimal. YMMV of course - all depends on the memory allocation 
 patterns of the application.


 I understand your points for using a separate memory manager, and
 I agree with you that having less active allocations make for faster
 sweeps, no matter how little of them are scanned for pointers. However
 I just had an idea on how to implement generational collection on
 a non-moving GC which should solve your issues (and well, mines too)
 with the collector not being fast enough. I need to do some hacking on


I saw a paper about that. The idea was to simply have some list of
objects/pages in each generation and modify that lists instead of moving
objects. I can't remember the name of the paper so I can't find it now :S

The problem with generational collectors (in D) is that you need
read/write barriers to track inter-generational pointers (to be able to
use pointers to younger generations in the older ones as roots when
scanning), which can make the whole deal a little unpractical for
a language that doesn't want to impose performance penalty to thing you
wont use (I don't see a way to instrument read/writes to pointers to the
GC only). This is why RC was always rejected as an algorithm for the GC in
D, I think.

 my custom GC first, but I believe it could give yet another performance
 boost. I'll add my memory manager to my list of code modules to make
 public :)


-- 
Leandro Lucarella (luca) | Blog colectivo: http://www.mazziblog.com.ar/blog/
----------------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145  104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------------
Pack and get dressed
before your father hears us,
before all hell breaks loose.

Sep 14 2009

Fawzi Mohamed <fmohamed mac.com> writes:

On 2009-09-14 17:07:00 +0200, "Robert Jacques" <sandford jhu.edu> said:

 On Mon, 14 Sep 2009 09:39:51 -0400, Leandro Lucarella 
 <llucax gmail.com>  wrote:
 Jeremie Pelletier, el 13 de septiembre a las 22:58 me escribiste:


 I understand your points for using a separate memory manager, and
 I agree with you that having less active allocations make for faster
 sweeps, no matter how little of them are scanned for pointers. However
 I just had an idea on how to implement generational collection on
 a non-moving GC which should solve your issues (and well, mines too)
 with the collector not being fast enough. I need to do some hacking on


 I saw a paper about that. The idea was to simply have some list of
 objects/pages in each generation and modify that lists instead of moving
 objects. I can't remember the name of the paper so I can't find it now :S
 
 The problem with generational collectors (in D) is that you need
 read/write barriers to track inter-generational pointers (to be able to
 use pointers to younger generations in the older ones as roots when
 scanning), which can make the whole deal a little unpractical for
 a language that doesn't want to impose performance penalty to thing you
 wont use (I don't see a way to instrument read/writes to pointers to the
 GC only). This is why RC was always rejected as an algorithm for the GC  in
 D, I think.
 
 my custom GC first, but I believe it could give yet another performance
 boost. I'll add my memory manager to my list of code modules to make
 public :)




 As a counter-point, objective-c just introduced a thread-local GC.  
 According to a blog post  
 (http://www.sealiesoftware.com/blog/archive/2009/08/28/objc_explain_Thread-local_garb
ge_collection.html) 
  apparently this has allowed pause times similar to the pause times of 
 the  previous generational GC. (Except that the former is doing a full 
 collect,  and the later still has work to do) On that note, it would 
 probably be a  good idea if core.gc.BlkAttr supported shared and 
 immutable state flags,  which could be used to support a thread-local 
 GC.


1) to allocate large objects that have a guard object it is a good idea 
to pass through the GC because if memory is tight a gc collection is 
triggered thereby possibly freeing some extra memory
2) using gc malloc is not faster than malloc, especially with several 
threads the single lock of the basic gc makes itself felt.

for how I use D (not realtime) the two things I would like to see from 
new gc are:
1) multiple pools (at least one per cpu, with thread id hash to assign 
threads to a given pool).
This to avoid the need of a global gc lock in the gc malloc, and if 
possible use memory close to the cpu when a thread is pinned, not to 
have really thread local memory, if you really need local memory 
different from the stack then maybe a separate process should be used. 
This is especially well doable with 64 bits, with 32 memory 
usage/fragmentation could become an issue.
2) multiple thread doing the collection (a main thread distributing the 
work to other threads (one per cpu), that do the mark phase using 
atomic ops).

other better gc, less latency (but not at the cost of too much 
computation), would be nice to have, but are not a priority for my 
usage.

Fawzi

Sep 14 2009

Fawzi Mohamed <fmohamed mac.com> writes:

On 2009-09-15 04:51:19 +0200, "Robert Jacques" <sandford jhu.edu> said:

 On Mon, 14 Sep 2009 18:53:51 -0400, Fawzi Mohamed <fmohamed mac.com> wrote:
 
 On 2009-09-14 17:07:00 +0200, "Robert Jacques" <sandford jhu.edu> said:
 
 On Mon, 14 Sep 2009 09:39:51 -0400, Leandro Lucarella  
 <llucax gmail.com>  wrote:
 Jeremie Pelletier, el 13 de septiembre a las 22:58 me escribiste:




 idea  to pass through the GC because if memory is tight a gc collection 
 is  triggered thereby possibly freeing some extra memory
 2) using gc malloc is not faster than malloc, especially with several  
 threads the single lock of the basic gc makes itself felt.
 
 for how I use D (not realtime) the two things I would like to see from  
 new gc are:
 1) multiple pools (at least one per cpu, with thread id hash to assign  
 threads to a given pool).
 This to avoid the need of a global gc lock in the gc malloc, and if  
 possible use memory close to the cpu when a thread is pinned, not to  
 have really thread local memory, if you really need local memory  
 different from the stack then maybe a separate process should be used.  
 This is especially well doable with 64 bits, with 32 memory  
 usage/fragmentation could become an issue.
 2) multiple thread doing the collection (a main thread distributing the 
  work to other threads (one per cpu), that do the mark phase using 
 atomic  ops).
 
 other better gc, less latency (but not at the cost of too much  
 computation), would be nice to have, but are not a priority for my 
 usage.
 
 Fawzi
 


 For what it's worth, the whole point of thread-local GC is to do 1) and 
  2). For the purposes of clarity, thread-local GC refers to each thread 
  having it's own GC for non-shared objects + a shared GC for shared  
 objects. Each thread's GC may allocate and collect independently of 
 each  other (e.g. in parallel) without locking/atomics/etc.


Well I want at least thread local pools (or almost, one can probably 
restrict it to the number of cpus, which will give most of the 
benefit), but not an extra partition of the memory in thread local and 
shared.
Such a partition might be easier in D2 (I think it was discussed, but 
even then I am not fully sure about the benefit), because then you have 
to somehow be able to share and maybe even unshare an object, which 
will be cumbersome. Thread local things add a level in the memory 
hierarchy that I am not fully sure is worth having, in it you should 
have almost only low level plumbing.
If you really want that much separation for many things then maybe a 
separate process + memmap might be better.
The fast local storage for me is the stack, and one might think about 
being more aggressive in using it, the heap is potentially shared.
Well at least that is my feeling.

Note that on 64 bit one can easily use a few bits to subdivide the 
memory in parts, making finding the pool group very quick, and this 
discussion is orthogonal to being generational or not.

Fawzi

Sep 15 2009

Jeremie Pelletier <jeremiep gmail.com> writes:

Fawzi Mohamed Wrote:

 On 2009-09-15 04:51:19 +0200, "Robert Jacques" <sandford jhu.edu> said:
 
 On Mon, 14 Sep 2009 18:53:51 -0400, Fawzi Mohamed <fmohamed mac.com> wrote:
 
 On 2009-09-14 17:07:00 +0200, "Robert Jacques" <sandford jhu.edu> said:
 
 On Mon, 14 Sep 2009 09:39:51 -0400, Leandro Lucarella  
 <llucax gmail.com>  wrote:
 Jeremie Pelletier, el 13 de septiembre a las 22:58 me escribiste:




 idea  to pass through the GC because if memory is tight a gc collection 
 is  triggered thereby possibly freeing some extra memory
 2) using gc malloc is not faster than malloc, especially with several  
 threads the single lock of the basic gc makes itself felt.
 
 for how I use D (not realtime) the two things I would like to see from  
 new gc are:
 1) multiple pools (at least one per cpu, with thread id hash to assign  
 threads to a given pool).
 This to avoid the need of a global gc lock in the gc malloc, and if  
 possible use memory close to the cpu when a thread is pinned, not to  
 have really thread local memory, if you really need local memory  
 different from the stack then maybe a separate process should be used.  
 This is especially well doable with 64 bits, with 32 memory  
 usage/fragmentation could become an issue.
 2) multiple thread doing the collection (a main thread distributing the 
  work to other threads (one per cpu), that do the mark phase using 
 atomic  ops).
 
 other better gc, less latency (but not at the cost of too much  
 computation), would be nice to have, but are not a priority for my 
 usage.
 
 Fawzi
 


 For what it's worth, the whole point of thread-local GC is to do 1) and 
  2). For the purposes of clarity, thread-local GC refers to each thread 
  having it's own GC for non-shared objects + a shared GC for shared  
 objects. Each thread's GC may allocate and collect independently of 
 each  other (e.g. in parallel) without locking/atomics/etc.


 Well I want at least thread local pools (or almost, one can probably 
 restrict it to the number of cpus, which will give most of the 
 benefit), but not an extra partition of the memory in thread local and 
 shared.
 Such a partition might be easier in D2 (I think it was discussed, but 
 even then I am not fully sure about the benefit), because then you have 
 to somehow be able to share and maybe even unshare an object, which 
 will be cumbersome. Thread local things add a level in the memory 
 hierarchy that I am not fully sure is worth having, in it you should 
 have almost only low level plumbing.
 If you really want that much separation for many things then maybe a 
 separate process + memmap might be better.
 The fast local storage for me is the stack, and one might think about 
 being more aggressive in using it, the heap is potentially shared.
 Well at least that is my feeling.
 
 Note that on 64 bit one can easily use a few bits to subdivide the 
 memory in parts, making finding the pool group very quick, and this 
 discussion is orthogonal to being generational or not.
 
 Fawzi
 


I just posted my memory manager to pastebin:
http://pastebin.com/f7459ba9d

I gave up on the generational feature, its indeed impossible without write
barriers to keep track of pointers from old generations to newer ones. I had
the whole tracing algorithm done but without generations, a naive scan and
sweep is faster because it has way less cache misses.

I'd like to get some feedback on it if possible.

Sep 16 2009

Lutger <lutger.blijdestijn gmail.com> writes:

Jeremie Pelletier wrote:
...
 
 I just posted my memory manager to pastebin:
 http://pastebin.com/f7459ba9d
 
 I gave up on the generational feature, its indeed impossible without write
 barriers to keep track of pointers from old generations to newer ones. I
 had the whole tracing algorithm done but without generations, a naive scan
 and sweep is faster because it has way less cache misses.
 
 I'd like to get some feedback on it if possible.


I think that it deserves a new thread...

Sep 16 2009

dsimcha <dsimcha yahoo.com> writes:

== Quote from Lutger (lutger.blijdestijn gmail.com)'s article
 Jeremie Pelletier wrote:
 ...
 I just posted my memory manager to pastebin:
 http://pastebin.com/f7459ba9d

 I gave up on the generational feature, its indeed impossible without write
 barriers to keep track of pointers from old generations to newer ones. I
 had the whole tracing algorithm done but without generations, a naive scan
 and sweep is faster because it has way less cache misses.

 I'd like to get some feedback on it if possible.




Yes, preferably on D.announce, and please explain what you did for the people
who
didn't read the original (horribly long, off-topic) thread.

Sep 16 2009

Lutger <lutger.blijdestijn gmail.com> writes:

Jeremie Pelletier wrote:

 Tom S Wrote:


 Sweet :D As for a place, there are plenty of options, e.g.
 http://dsource.org/projects/scrapple/ or a separate dsource project.


 I thought of that, but I don't feel like opening a project for just a few
 random code snippets or standalone classes. I think I'll just post it in
 this forum and let interested people grab it for now.


That's cool, but scrapple is exactly that: an assortment of small(ish)  
projects / pieces of code that otherwise don't warrant a full project. If 
you feel like putting it online, just ping BCS and I'm sure he'll give you 
access right away.

Stacktrace is a feature highly desired by lots of people, you will many 
developers happy :)

Sep 13 2009

BCS <none anon.com> writes:

Hello Lutger,
 That's cool, but scrapple is exactly that: an assortment of small(ish)
 projects / pieces of code that otherwise don't warrant a full project.
 If you feel like putting it online, just ping BCS and I'm sure he'll
 give you access right away.


All I need is your dsource user name.

Sep 14 2009

bearophile <bearophileHUGS lycos.com> writes:

Jeremie Pelletier:

I haven't had to use the C heap whatsoever so far in D, could you give me an
example of where you need it?<


1) I'm able to allocate a bigger single chunk of memory from the C heap, about
1.8 GB, while the GC heap of DMD on Windows allows only for a smaller chunk. In
a program I've had to allocate a single chunk of memory.
2) I think the GC is faster at allocating small chunks of memory, while the C
heap is faster at allocating large chunks.
3) GC-managed pointers have several restrictions (so much different that maybe
I'd like them to be seen as a different type from the compiler, that requires a
cast to be converted from/to C pointers. I don't know why this idea was not
appreciated by D designers). One of them is that GC-managed pointers can't be
tagged, you can't add one or two bits to GC-managed pointers, and such tags are
useful when you want to implement certain data structures.


Indeed, and sometimes it's way faster than that.<


But lot of people will judge D against more modern languages like C#, Scala 8or
Java) and not against C.


C++ isn't anymore complex than D2,<


I don't agree, see below.


I can't think of many features C++ has over D2. I can name quite a few features
D2 has over C++ :)<


Complexity and not-simplicity come from many things, like corner cases, rules
against rules against rules, unsafe things that the compiler isn't able to
catch in case of programmer errors, unnatural syntax, and not just from
features. For example the Python 3+ language has many more features than pure
C, yet Python3 is simpler than C :-)


it still allows for dirty work to be done when you need it.<


The less dirty you make it the better it will be when you try to mantain/debug
your D code :-)


You don't need to code your application core in C and your application behavior
in a scripting language on top of the C core. D allows you to write it all in
one language with the same productivity, if not better productivity for not
having to write the abstraction layer between C and scripting.<


While D is quite better than C, that of yours is a dream. In practice D isn't
dynamic and for certain purposes D is not close to the "productivity" of Python
:-) (Even just because in Python you can find tons of modules and bindings
already done). There are ways to improve D still for such purposes. D can be
more scalable as Scala :-)

Bye,
bearophile

Sep 14 2009

Lutger <lutger.blijdestijn gmail.com> writes:

language_fan wrote:

 Mon, 14 Sep 2009 07:33:59 -0400, bearophile thusly wrote:
 
 But lot of people will judge D against more modern languages like C#,
 Scala or Java) and not against C.


 Programmers often belong to three kinds of groups. First come the fans of
 traditionally weakly typed compiled languages (basic, c, c++). They have
 tried some "dynamic" or "academic" languages but did not like them. They
 fancy efficiency and close to metal feel. They think compilation to
 native code is the best way to produce programs, and think types should
 reflect the feature set of their cpu. They believe the syntax C uses was
 defined by their God.
 
 The second group started with interpreted languages built by amateurs
 (php, ruby, python, some game scripting language etc). They do not
 understand the meaning the types or compilation. They prefer writing
 short programs that usually seem to work. They hate formal specifications
 and proofs about program properties. They are usually writing simple web
 applications or some basic shareware utilies no one uses. They also hate
 trailing semicolons.
 
 The members of the last group have studied computer science and
 languages, in particular. They have found a pet academic language,
 typically a pure one, but paradigms may differ. In fact this is the group
 which uses something other than the hybrid object-oriented/procedural
 model. They appreciate a strong, orthogonal core language that scales
 cleanly. They are not scared of esoteric non-C-like syntax. They use
 languages that are not ready to take a step to the "real world" during
 the 70 next years.
 


That's a fancy way of saying that anyone who has not studied CS is a moron 
and therefore cannot understand what is good about languages, thus they lose 
any argument automatically. Am I right?

Sep 14 2009

Jeremie Pelletier <jeremiep gmail.com> writes:

Lutger Wrote:

 language_fan wrote:
 
 Mon, 14 Sep 2009 07:33:59 -0400, bearophile thusly wrote:
 
 But lot of people will judge D against more modern languages like C#,
 Scala or Java) and not against C.


 Programmers often belong to three kinds of groups. First come the fans of
 traditionally weakly typed compiled languages (basic, c, c++). They have
 tried some "dynamic" or "academic" languages but did not like them. They
 fancy efficiency and close to metal feel. They think compilation to
 native code is the best way to produce programs, and think types should
 reflect the feature set of their cpu. They believe the syntax C uses was
 defined by their God.
 
 The second group started with interpreted languages built by amateurs
 (php, ruby, python, some game scripting language etc). They do not
 understand the meaning the types or compilation. They prefer writing
 short programs that usually seem to work. They hate formal specifications
 and proofs about program properties. They are usually writing simple web
 applications or some basic shareware utilies no one uses. They also hate
 trailing semicolons.
 
 The members of the last group have studied computer science and
 languages, in particular. They have found a pet academic language,
 typically a pure one, but paradigms may differ. In fact this is the group
 which uses something other than the hybrid object-oriented/procedural
 model. They appreciate a strong, orthogonal core language that scales
 cleanly. They are not scared of esoteric non-C-like syntax. They use
 languages that are not ready to take a step to the "real world" during
 the 70 next years.
 


 That's a fancy way of saying that anyone who has not studied CS is a moron 
 and therefore cannot understand what is good about languages, thus they lose 
 any argument automatically. Am I right?
 


I dunno if that's what OP meant, but studying CS does not make you a reference
in programming languages. I didn't even complete my first year of CS because I
wasn't learning as fast as I wanted. School teaches you theory anyways, a job
will teach you how to apply it in the real world. Anyone who can read and has
the slightest interest in programming can learn the theory by themselves.

As for the different classes of programmers, I think the OP pushed more the
extremes than the general cases. I came across a series of articles by Eric
Lippert a few weeks ago talking about the matter:

http://blogs.msdn.com/ericlippert/archive/tags/Cargo+Cult+Programming/default.aspx

Sep 14 2009

Don <nospam nospam.com> writes:

language_fan wrote:
 Tue, 15 Sep 2009 00:25:46 +0200, Lutger thusly wrote:
 
 That's a fancy way of saying that anyone who has not studied CS is a
 moron and therefore cannot understand what is good about languages, thus
 they lose any argument automatically. Am I right?


 I just recommend learning basic concepts until terms like generational 
 garbage collection, closure, register allocation, immutability, loop 
 fusion, term rewriting, regular languages, type constructor, virtual 
 constructor, and covariance do not scare you anymore.
 
 If something small like optional semicolons or some other syntactic 
 nuance prevents you from finishing your job, how the heck are you 
 supposed to build any real world programs? Just to put this to some 
 perspective, syntax matters, but not much. 


Nowadays you can easily write
 a tool that parses stuff written in language X and outputs it in pretty 
 printed form in language Y. This is what happens on .NET, for instance. 
 Most of the languages there are just syntactic skins for the same common 
 core language.


It sounds as though talking about VB.NET, which is a non-existent 
language (it's a parsing step ONLY). It's just C# with a different parse 
table, and exists only for marketing reasons (to disguise the fact that 
MS abandoned VB). I don't think you can conclude anything general from that.

Sep 15 2009

Justin Johansson <procode adam-dott-com.au> writes:

language_fan Wrote:

 ... Nowadays you can easily write 
 a tool that parses stuff written in language X and outputs it in pretty 
 printed form in language Y. This is what happens on .NET, for instance. 
 Most of the languages there are just syntactic skins for the same common 
 core language.


Yes, well, Ted Neward pretty much makes this observation re LLVM saying

"Holy frickin' crap. I think I'm in love."

In case you missed it, here's the link

http://blogs.tedneward.com/2008/02/24/Some+Interesting+Tidbits+About+LLVM.aspx

<JJ/>

Sep 15 2009

Lutger <lutger.blijdestijn gmail.com> writes:

language_fan wrote:

 Tue, 15 Sep 2009 00:25:46 +0200, Lutger thusly wrote:
 
 That's a fancy way of saying that anyone who has not studied CS is a
 moron and therefore cannot understand what is good about languages, thus
 they lose any argument automatically. Am I right?


 I just recommend learning basic concepts until terms like generational
 garbage collection, closure, register allocation, immutability, loop
 fusion, term rewriting, regular languages, type constructor, virtual
 constructor, and covariance do not scare you anymore.
 


Right right, I don't disagree with that. It was more the 'ruby/python 
programmers make apps no-one uses using amateur tools | c-family users 
worship FOO and the rest are academics that use pure functional languages' 
part that tripped me up. You know, the majority of software isn't built by 
academics, NASA uses C mostly, etc. A little nuance wouldn't hurt here.

Sep 15 2009

Christopher Wright <dhasenan gmail.com> writes:

language_fan wrote:
 In fact this is the group 
 which uses something other than the hybrid object-oriented/procedural 
 model.


Damn straight! They use a hybrid OO/procedural/functional model. Like D. 
Or C#.

"Oh, but Prolog," you may say. And I admit, I've seen it used a couple 
times by academics. But I've seen similar languages used at my job, and 
we're not by any means an algorithms shop.

I'm actually aware of very few languages that break the mold. There are 
toy languages like Befunge; there are solver-oriented languages like 
Prolog and Zimpl; and there are a couple oddities like METAFONT.

What languages have you seen that are so innovative and different in 
paradigm?

Sep 14 2009

Rainer Deyke <rainerd eldwood.com> writes:

language_fan wrote:
 The members of the last group have studied computer science and 
 languages, in particular. They have found a pet academic language, 
 typically a pure one, but paradigms may differ. In fact this is the group 
 which uses something other than the hybrid object-oriented/procedural 
 model. They appreciate a strong, orthogonal core language that scales 
 cleanly. They are not scared of esoteric non-C-like syntax. They use 
 languages that are not ready to take a step to the "real world" during 
 the 70 next years.


Of the three types, this comes closest to describing me.  Yet, I am
completely self-taught, and my preferred language is still C++.  (I
wouldn't call it my pet language.  I loathe C++, I just haven't found a
suitable replacement yet.)

Stereotypes are dangerous.


-- 
Rainer Deyke - rainerd eldwood.com

Sep 14 2009

"Robert Jacques" <sandford jhu.edu> writes:

On Mon, 14 Sep 2009 09:39:51 -0400, Leandro Lucarella <llucax gmail.com>  
wrote:
 Jeremie Pelletier, el 13 de septiembre a las 22:58 me escribiste:


 I understand your points for using a separate memory manager, and
 I agree with you that having less active allocations make for faster
 sweeps, no matter how little of them are scanned for pointers. However
 I just had an idea on how to implement generational collection on
 a non-moving GC which should solve your issues (and well, mines too)
 with the collector not being fast enough. I need to do some hacking on


 I saw a paper about that. The idea was to simply have some list of
 objects/pages in each generation and modify that lists instead of moving
 objects. I can't remember the name of the paper so I can't find it now :S

 The problem with generational collectors (in D) is that you need
 read/write barriers to track inter-generational pointers (to be able to
 use pointers to younger generations in the older ones as roots when
 scanning), which can make the whole deal a little unpractical for
 a language that doesn't want to impose performance penalty to thing you
 wont use (I don't see a way to instrument read/writes to pointers to the
 GC only). This is why RC was always rejected as an algorithm for the GC  
 in
 D, I think.

 my custom GC first, but I believe it could give yet another performance
 boost. I'll add my memory manager to my list of code modules to make
 public :)




As a counter-point, objective-c just introduced a thread-local GC.  
According to a blog post  
(http://www.sealiesoftware.com/blog/archive/2009/08/28/objc_explain_Thread-local_garb
ge_collection.html)  
apparently this has allowed pause times similar to the pause times of the  
previous generational GC. (Except that the former is doing a full collect,  
and the later still has work to do) On that note, it would probably be a  
good idea if core.gc.BlkAttr supported shared and immutable state flags,  
which could be used to support a thread-local GC.

Sep 14 2009

Rainer Deyke <rainerd eldwood.com> writes:

Nick Sabalausky wrote:
 That's *very* programmer-dependent. It originally took took me all of about 
 a week to get used to semicolons after growing up on basic (and even then it 
 was a very very minor time sink), and now just it takes all of about a split 
 second to press that key. But, any time I use a language that doesn't allow 
 semicolon line endings, I keep sticking them in without even thinking about 
 it. Then the compiler complains, and I have to go back and fix it, and that 
 slows down programming more than just instinctively hitting a key.


If you're going to judge features on the basis of habit, then the best
language is always the language you have been using for the longest time.

I'm not entirely happy with the way Scala handles the division between
statements - Scala's rules seem arbitrary and complex - but semicolons
*are* noise, no matter how habitually I use them and how much time I
waste removing them afterwards.

My preferred rule is this:  If two lines have the same indentation, they
are separate statements.  If the second line is indented further than
the first line, the second line is a continuation of the statement
started in the first line.  Surprisingly, even Python (which already has
significant indentation) doesn't use this simple and obvious rule.


-- 
Rainer Deyke - rainerd eldwood.com

Sep 11 2009

Benji Smith <dlanguage benjismith.net> writes:

Rainer Deyke wrote:
 I'm not entirely happy with the way Scala handles the division between
 statements - Scala's rules seem arbitrary and complex - but semicolons
 *are* noise, no matter how habitually I use them and how much time I
 waste removing them afterwards.


I don't know anything about scala, but I've been working on an 
Actionscript compiler recently (the language is based on ECMAScript, so 
it's very much like JavaScript in this respect) and the optional 
semicolon rules are completely maddening.

The ECMAScript spec basically says: virtual semicolons must be inserted 
at end-of-line whenever the non-insertion of semicolons would result in 
an erroneous parse.

So there are really only three ways to handle it, and all of them are 
insane:

1) Treat the newline character as a token (rather than as skippable 
whitespace) and include that token as an optional construct in every 
single production where it can legally occur. This results in hundreds 
of optional semicolons throughout the grammar, and makes the whole thing 
a nightmare to read, but at least it still uses a one-pass CFG.

     CLASS :=
       "class"
       NEWLINE?
       IDENTIFIER
       NEWLINE?
       "{"
       NEWLINE?
       (
         MEMBER
         NEWLINE?
       )*
       "}"

2) Use lexical lookahead, dispatched from the parser. The tokenizer 
determines whether to treat a newline as a statement terminator based on 
the current parse state (are we in the middle of a parenthetized 
expression?) and the upcoming tokens on the next line. This is nasty 
because the grammar becomes context-sensitive and conflates lexical 
analysis with parsing.

2) Whenever the parser encounters an error, have it back up to the 
beginning of the previous production and insert a virtual semicolon into 
the token stream. Then try reparsing. Since there might be multiple 
newlines contained in a single multiline expression, it might take 
arbitrarily many rewrite attempts before reaching a correct parse.

The thing about most compiler construction tools is that they don't 
allow interaction between the context-guided tokenization, and they're 
not designed for the creation of backup-and-retry processing, or the 
insertion of virtual tokens into the token stream.

Ugly stuff.

Anyhoo, I know this is waaaaaaay off topic. But I think any language 
designer including optional semicolons in their language desperately 
deserves a good swift punch in the teeth.

--benji

Sep 11 2009

Rainer Deyke <rainerd eldwood.com> writes:

Benji Smith wrote:
 The ECMAScript spec basically says: virtual semicolons must be inserted
 at end-of-line whenever the non-insertion of semicolons would result in
 an erroneous parse.


And I don't like that rule, and I've proposed a sensible alternative, but...

 So there are really only three ways to handle it, and all of them are
 insane:


 
 1) Treat the newline character as a token (rather than as skippable
 whitespace) and include that token as an optional construct in every
 single production where it can legally occur. This results in hundreds
 of optional semicolons throughout the grammar, and makes the whole thing
 a nightmare to read, but at least it still uses a one-pass CFG.
 
     CLASS :=
       "class"
       NEWLINE?
       IDENTIFIER
       NEWLINE?
       "{"
       NEWLINE?
       (
         MEMBER
         NEWLINE?
       )*
       "}"


4) Treat newlines as tokens.  Whenever a non-newline token appears in
the grammar and a newline token appears in the token stream, skip an
arbitrary number of newline tokens to get to the next non-newline token.

This is equivalent to replacing every non-newline terminal symbol X in
the grammar with the (NEWLINE* X) sequence.  You obviously wouldn't do
that by hand.  You've got a general purpose computer in front of you.
Use it!

 The thing about most compiler construction tools is that they don't
 allow interaction between the context-guided tokenization, and they're
 not designed for the creation of backup-and-retry processing, or the
 insertion of virtual tokens into the token stream.


And don't use outdated crappy compiler construction tools.


-- 
Rainer Deyke - rainerd eldwood.com

Sep 11 2009

language_fan <foo bar.com.invalid> writes:

Mon, 14 Sep 2009 07:33:59 -0400, bearophile thusly wrote:

 But lot of people will judge D against more modern languages like C#,
 Scala or Java) and not against C.


Programmers often belong to three kinds of groups. First come the fans of 
traditionally weakly typed compiled languages (basic, c, c++). They have 
tried some "dynamic" or "academic" languages but did not like them. They 
fancy efficiency and close to metal feel. They think compilation to 
native code is the best way to produce programs, and think types should 
reflect the feature set of their cpu. They believe the syntax C uses was 
defined by their God.

The second group started with interpreted languages built by amateurs 
(php, ruby, python, some game scripting language etc). They do not 
understand the meaning the types or compilation. They prefer writing 
short programs that usually seem to work. They hate formal specifications 
and proofs about program properties. They are usually writing simple web 
applications or some basic shareware utilies no one uses. They also hate 
trailing semicolons.

The members of the last group have studied computer science and 
languages, in particular. They have found a pet academic language, 
typically a pure one, but paradigms may differ. In fact this is the group 
which uses something other than the hybrid object-oriented/procedural 
model. They appreciate a strong, orthogonal core language that scales 
cleanly. They are not scared of esoteric non-C-like syntax. They use 
languages that are not ready to take a step to the "real world" during 
the 70 next years.

So yes, every group has a bit different expectations..

C++ isn't anymore complex than D2,<


 I don't agree, see below.


It would help all of you if you could somehow formally specify how you 
measure language complexity. Is it the length of the grammar definition 
or something else? Otherwise these are just subjective opinions.

Sep 14 2009

language_fan <foo bar.com.invalid> writes:

Mon, 14 Sep 2009 21:55:10 -0600, Rainer Deyke thusly wrote:

 language_fan wrote:
 The members of the last group have studied computer science and
 languages, in particular. They have found a pet academic language,
 typically a pure one, but paradigms may differ. In fact this is the
 group which uses something other than the hybrid
 object-oriented/procedural model. They appreciate a strong, orthogonal
 core language that scales cleanly. They are not scared of esoteric
 non-C-like syntax. They use languages that are not ready to take a step
 to the "real world" during the 70 next years.


 Of the three types, this comes closest to describing me.  Yet, I am
 completely self-taught, and my preferred language is still C++.  (I
 wouldn't call it my pet language.  I loathe C++, I just haven't found a
 suitable replacement yet.)
 
 Stereotypes are dangerous.


Indeed they are. My post should have been taken with a grain of salt. The 
idea was to show that languages in each group have their advantages and 
disadvantages. There is nothing wrong with being self-taught, many times 
people with formal education lack the passion many amateurs share.

What is bad is that many people can only express their ideas in one kind 
of language, and that is usually their pet language. If you study Java, 
C#, C++, and D, they are all very similar to each other. Especially if 
you try to avoid learning all advanced features that are not common to 
all of them. In that case you don't know four different languages, but a 
single simple language mostly suitable for basic end user application 
development. On the other hand, knowing 40 academic languages will not 
get you far, either.

Sep 15 2009

language_fan <foo bar.com.invalid> writes:

Tue, 15 Sep 2009 00:25:46 +0200, Lutger thusly wrote:

 That's a fancy way of saying that anyone who has not studied CS is a
 moron and therefore cannot understand what is good about languages, thus
 they lose any argument automatically. Am I right?


I just recommend learning basic concepts until terms like generational 
garbage collection, closure, register allocation, immutability, loop 
fusion, term rewriting, regular languages, type constructor, virtual 
constructor, and covariance do not scare you anymore.

If something small like optional semicolons or some other syntactic 
nuance prevents you from finishing your job, how the heck are you 
supposed to build any real world programs? Just to put this to some 
perspective, syntax matters, but not much. Nowadays you can easily write 
a tool that parses stuff written in language X and outputs it in pretty 
printed form in language Y. This is what happens on .NET, for instance. 
Most of the languages there are just syntactic skins for the same common 
core language.

Sep 15 2009

language_fan <foo bar.com.invalid> writes:

Tue, 15 Sep 2009 16:41:25 +0200, Lutger thusly wrote:

 language_fan wrote:
 
 Tue, 15 Sep 2009 00:25:46 +0200, Lutger thusly wrote:
 
 That's a fancy way of saying that anyone who has not studied CS is a
 moron and therefore cannot understand what is good about languages,
 thus they lose any argument automatically. Am I right?


 I just recommend learning basic concepts until terms like generational
 garbage collection, closure, register allocation, immutability, loop
 fusion, term rewriting, regular languages, type constructor, virtual
 constructor, and covariance do not scare you anymore.
 
 


 programmers make apps no-one uses using amateur tools | c-family users
 worship FOO and the rest are academics that use pure functional
 languages' part that tripped me up. You know, the majority of software
 isn't built by academics, NASA uses C mostly, etc. A little nuance
 wouldn't hurt here.


Ok. I did not even mean you should write all your programs in LISP or 
Prolog. The point is, once you know how to use various kinds of 
techniques, you can use whatever language you want. The problem is, most 
programmers only know 1-3 languages, and those languages are typically 
very similar to each other (e.g. Java, C, and C++). Some problems are 
inherently functional or easily expressed with regular expressions or as 
a logical constraint satisfaction problem. It does not make sense to 
write your own priority queue for each new task.

Sep 15 2009

"Robert Jacques" <sandford jhu.edu> writes:

On Mon, 14 Sep 2009 18:53:51 -0400, Fawzi Mohamed <fmohamed mac.com> wrote:

 On 2009-09-14 17:07:00 +0200, "Robert Jacques" <sandford jhu.edu> said:

 On Mon, 14 Sep 2009 09:39:51 -0400, Leandro Lucarella  
 <llucax gmail.com>  wrote:
 Jeremie Pelletier, el 13 de septiembre a las 22:58 me escribiste:


 I understand your points for using a separate memory manager, and
 I agree with you that having less active allocations make for faster
 sweeps, no matter how little of them are scanned for pointers. However
 I just had an idea on how to implement generational collection on
 a non-moving GC which should solve your issues (and well, mines too)
 with the collector not being fast enough. I need to do some hacking on


 objects/pages in each generation and modify that lists instead of  
 moving
 objects. I can't remember the name of the paper so I can't find it now  
 :S
  The problem with generational collectors (in D) is that you need
 read/write barriers to track inter-generational pointers (to be able to
 use pointers to younger generations in the older ones as roots when
 scanning), which can make the whole deal a little unpractical for
 a language that doesn't want to impose performance penalty to thing you
 wont use (I don't see a way to instrument read/writes to pointers to  
 the
 GC only). This is why RC was always rejected as an algorithm for the  
 GC  in
 D, I think.

 my custom GC first, but I believe it could give yet another  
 performance
 boost. I'll add my memory manager to my list of code modules to make
 public :)




 According to a blog post   
 (http://www.sealiesoftware.com/blog/archive/2009/08/28/objc_explain_Thread-local_garb
ge_collection.html)  
  apparently this has allowed pause times similar to the pause times of  
 the  previous generational GC. (Except that the former is doing a full  
 collect,  and the later still has work to do) On that note, it would  
 probably be a  good idea if core.gc.BlkAttr supported shared and  
 immutable state flags,  which could be used to support a thread-local  
 GC.


 1) to allocate large objects that have a guard object it is a good idea  
 to pass through the GC because if memory is tight a gc collection is  
 triggered thereby possibly freeing some extra memory
 2) using gc malloc is not faster than malloc, especially with several  
 threads the single lock of the basic gc makes itself felt.

 for how I use D (not realtime) the two things I would like to see from  
 new gc are:
 1) multiple pools (at least one per cpu, with thread id hash to assign  
 threads to a given pool).
 This to avoid the need of a global gc lock in the gc malloc, and if  
 possible use memory close to the cpu when a thread is pinned, not to  
 have really thread local memory, if you really need local memory  
 different from the stack then maybe a separate process should be used.  
 This is especially well doable with 64 bits, with 32 memory  
 usage/fragmentation could become an issue.
 2) multiple thread doing the collection (a main thread distributing the  
 work to other threads (one per cpu), that do the mark phase using atomic  
 ops).

 other better gc, less latency (but not at the cost of too much  
 computation), would be nice to have, but are not a priority for my usage.

 Fawzi


For what it's worth, the whole point of thread-local GC is to do 1) and  
2). For the purposes of clarity, thread-local GC refers to each thread  
having it's own GC for non-shared objects + a shared GC for shared  
objects. Each thread's GC may allocate and collect independently of each  
other (e.g. in parallel) without locking/atomics/etc.

Sep 14 2009

D Programming

C/C++ Programming

Other

digitalmars.D - Re: Template Metaprogramming Made Easy (Huh?)