www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - How can D become adopted at my company?

reply "Eljay" <eljay451 gmail.com> writes:
[I had sent Walter an email, answering a question he asked me a 
few years ago.  Walter asked me to post the message here.  
Slightly redacted - I took out an non-relevant aside, and since 
the forum doesn't support HTML bold, italic, underscore tags 
those have been stripped as well.]

Hi Walter,

You had asked me a question that I have been thinking about for a 
couple years.

Your question from 2009 was, “How can D become adopted at my 
company?”


*** What makes a language successful? ***

I think Stroustrup’s answer is the best metric I’ve heard:  a 
language is successful if it is used.  Or in my wording, if you 
know that technology can you get a job utilizing that technology.

Looking at all the successful languages, I have noticed that all 
the successful ones I am familiar with have had some sort of 
sponsor pushing the technology.  For example, Ada by the U.S. 
government, C# by Microsoft, Objective-C by Apple, C and C++ by 
AT&T, Lua by PCU in Rio, REXX by IBM, the list goes on and on and 
on.

The only exception I can think of is Ruby, a successful language 
that prospered by a grassroots movement without a corporate, 
government, or academic sponsor as far as I’m aware.

My understanding is that Facebook is sponsoring D.  At least in 
some capacity, I’m not sure of the details or extent.  But 
still, it’s a start, and important, and Facebook has monetary 
resources.


*** What makes a language popular? ***

The classic chicken-and-egg problem.  A language is popular 
because a lot of people use it.  A lot of people use it because 
it is popular.

So how does a budding language like D become more popular?  
That’s a marketing and evangelist problem.  And not my forte.  
But I have raised awareness of D with developers I know, 
word-of-mouth.

Some leading-edge independent developers have used D, and their 
programs demonstrate that D can walk-the-walk as well as 
talk-the-talk (e.g., Torus Troopers using D 0.110 and OpenGL).

Maybe Facebook could be convinced to pay some book writers to 
make D books.
    * Numerical Recipes in D
    * Design Patterns in D
    * The Dao of D
    * Harry Potter and the Frumious D Compiler


*** Who will use D?  And why?  And for what? ***

D is a neat language because it is a systems programming 
language, suitable as a more-than-viable alternative to C or C++.

But unlike C and C++, it has Safe D, which is eminently suitable 
for applications programming language.  Sure, C and C++ are used 
as applications programming languages, but that’s not what they 
were designed for.  D has an applications language story that is 
far more compelling than C and C++, and has the potential as a 
native applications programming language that VM based languages 
can’t touch.


*** What does D lack? ***

I think it is interesting to take a step back, and look at the 
whole enchilada of programming.  From Python to BASIC, from SQL 
to Brainf**k.

I think all these following things could be asked for as D 
Project Requests to the D user community.

Web framework.  What do people use Ruby on Rails, or slightly 
less popular Python and Django for?  Is D a suitable alternative? 
  Maybe, maybe not.  D can do the compiling job, no doubt, but is 
there a “Rails” or “Django” for D that is as strong as, 
well, Rails or Django?

Scripting.  Many games use Lua as an embedded scripting language. 
  With mixin we could write our own DSL in D.  Possibly even 
re-implement Lua-in-D.  Yet, I think D could greatly benefit by 
coming with a canned scripting language (e.g., a Lua-like 
language), as a standard feature.  Perhaps someone will do so (or 
has done so), and submit it for consideration.  That would make 
plumbing up a do-it-yourself scripting language (like Lua-on-C) 
to your engine-written-in-D so much easier.  If not Lua-in-D, 
perhaps JavaScript-in-D...?

Linux kernel in D.  This would be a crazy project for someone who 
is a Linux lover, an operating systems wonk, a D enthusiast, and 
highly OCD.  It is crazy because is a lot of effort to re-write 
something like a Linux kernel in another language, because it is 
a lot of work without any visible gain at the end.  But the final 
result would showcase that D can do the heavy lifting of an 
operating system.  Let’s see /language-X/ do that!  (Where 
/language-X/ is not Assembly, C, C++, or Sing#.)

D on iOS.  So for me personally, I would love to use D to write 
my applications for iOS, and OS X.  But… I’m not sure how to 
do that.  (The Objective-D project looks abandoned, never got out 
of the “toy” project stage, and doesn’t bridge Cocoa’s 
Frameworks written in Objective-C to D/Objective-D anyway.)

D for .NET.  Or I would love to use D to write my applications 
for .NET … and again, I’m not sure how to do that.  (The 
D.NET project looks abandoned, and never got out of the “toy” 
project stage.  And I don’t see a way to generate CIL from D.)

(For the above two D Project Requests, I have to admit I 
haven’t really looked all that hard.  So they could be already 
solved problems that I’m just ignorant of the existing 
solution.)


*** What features would you add to D? ***

I think these two features would help D tremendously:
    * D comes with a canned embedded scripting language, like Lua
    * D comes with the Objective facilities, like those found in 
Objective-C

For embedded scripting languages, Lua has shown it has the right 
balance of tiny-tiny-tiny footprint, rich expressivity, and 
highly performant.  It has been used in many games like World of 
Warcraft, and desktop applications such as the logic-and-UI 
language for Adobe Lightroom.

So why not just use Lua itself in D?  Because Lua has a nice 
Lua-to-C API, and in my opinion having Lua-in-D would be able to 
leverage some of D’s strengths and make the scripting language 
seamless.

The Objective portion of Objective-C is very interesting.  The 
amazing advantage of Objective-Whatever is not in the Whatever 
inasmuch as it is in what the Objective portion brings to the 
table.  Since all Objective objects use message-and-dispatch 
mechanism, it means that frameworks all become incredibly loosely 
coupled.  Also do to late-binding and the dispatch mechanism, 
anyone can extend any class or proxy any class or remote proxy 
any class, easily, even without source code.

To illustrate, I will compare C++ to Objective-C.  In C++, if you 
have a public API that takes in a std∷string const& as a 
parameter, you will soon discover that the std∷string const& is 
intimately affected by the compiler used and the optimization 
settings.  In Objective-C, the Objective string objects can be 
mixed-and-matched from different frameworks in which different 
string objective have entirely different implementations, but all 
comply with the same message contract.  And any framework can 
extend all the string objects used with novel functionality.  
That de-coupling is super-important for scalability, including 
plug-ins and extension frameworks.


*** What’s the future of programming? ***

The “Next Big Thing” for computer languages probably won’t 
be emphasizing their super-awesome encapsulation and way-cool 
message-and-dispatcher based de-coupling.  :-)

I think the Next Big Thing in computer languages is rich DSL 
support, which will enable more complexity by simplifying what 
needs to be written in a more suitably (i.e., domain specific) 
expressivity.  Due to mixin and generational programming, I think 
D enables DSL grammars really-really-really well.

Also, due to a confluence of factors, the rising star for 
becoming the most widely used programming language is JavaScript. 
  I have to say, I’m not a fan of JavaScript.  I’ve seen the 
leading edge of compile-to-JavaScript languages, such as 
CoffeeScript and DART.  Can D get on that bandwagon and have the 
D compiler compile to JavaScript as some sort of IL?  I know that 
sounds like utter crazy talk, like Script# (C# to JavaScript 
compiler).

Sincerely,
Eljay
Apr 24 2012
next sibling parent reply Trass3r <un known.com> writes:
 My understanding is that Facebook is sponsoring D.  At least in some  =

 capacity, I=E2=80=99m not sure of the details or extent.  But still, i=

 start, and important, and Facebook has monetary resources.

Andrei works at facebook, that's all.
 Web framework.  What do people use Ruby on Rails, or slightly less  =

 popular Python and Django for?  Is D a suitable alternative?

See Adam Ruppe's work.
 Scripting.  Many games use Lua as an embedded scripting language.   Wi=

 mixin we could write our own DSL in D.  Possibly even re-implement  =

 Lua-in-D.

No need to reinvent the wheel.
 So why not just use Lua itself in D?  Because Lua has a nice Lua-to-C =

 API, and in my opinion having Lua-in-D would be able to leverage some =

 D=E2=80=99s strengths and make the scripting language seamless.

LuaD.
 kernel in D.

XOmB.
 Can D get on that bandwagon and have the D compiler compile to  =

 JavaScript as some sort of IL?

Theoretically. See emscripten.
Apr 24 2012
parent "Eljay" <eljay451 gmail.com> writes:
Awesome!  Thanks Trass3r!
Apr 24 2012
prev sibling next sibling parent reply "Eljay" <eljay451 gmail.com> writes:
As a follow up to my email to Walter...

I know I didn't address the question "How can D become adopted at 
my company?" head-on.

An on-going project written in (say) C++ is not going to get 
approval to re-write in D.  There is no ROI in it.

A new project that could be written in D will be met with a lot 
of resistance.  Management will consider D too risky, as compared 
to writing the same project in C++ or C# or Java.  Co-workers not 
familiar with D will consider it as a pain-in-the-learning-curve 
[an attitude I cannot fathom; learning a new computer language is 
a joy, like opening a birthday present].

In some cases, such as shipping an application for iOS or Windows 
Phone or Android devices, can D even be utilized?  Even if 
management and the team's developers are behind using D?

---

A brief blurb about who I am...

I started programming in 1976, where I contributed to a program 
called Oregon Trail written in HP2000A BASIC on TIES.  That was 
my very first programming experience.

After learning BASIC, I learned 6502 assembly, then later picked 
up FORTRAN, Pascal, and C.  Then 68000 assembly.

I abandoned programming in assembly when I got my first 
optimizing C compiler, which was able to out-optimize my lovingly 
hand-crafted assembly.  I became a true believer in the powerful 
mojo of optimizing compilers.

In 1990, I switched from C to C++, first as as "Better C" 
compiler.  By two years later, I had fully embraced OOP style.

C++ was my main language for a long time, with a couple years 
doing Java.  Most recently, I have been programming in C#.

---

About 12 years ago, using Aho's dragon book by my side, I tried 
my hand at writing my own programming language.  After six 
months, I gave up because creating a good, general purpose 
programming language IS VERY VERY HARD.

Later, when I stumbled upon D, it was like Walter had read my 
mind and implemented what I could only conceive of... I was 
smitten.  And I still am.

So the languages I admire are...
    * D, as a general purpose natively compiled multi-paradigm 
programming language
    * Lua, as a barebones, small footprint, embed-able 
do-it-yourself scripting language
    * Python 3, as a kitchen-sink-included scripting language

I have used extensively BASIC (HP2000A, Apple Integer, Applesoft, 
MAI BusinessBASIC IV, PickBASIC), FORTRAN, Prolog, LISP & Scheme, 
6502 Assembly, 680x0 Assembly, Pascal, Mathematica, C, C++, 
Objective-C, Objective-C++, Java.

I'm also intrigued by some other languages but I do not use them 
day-to-day, such as F#, Spec#, Sing#.  And I certainly have toyed 
with many other programming languages, such as Perl, Ruby, REXX, 
Ada, Squeak, Forth, PostScript, yada yada yada.

My educational background is in high-energy physics where I 
learned FORTRAN, linguistics (with a focus on semantics and 
artificial intelligence) where I learned Prolog and LISP, and 
computer science.

---

And the most important bit of information:  I use vi (Vim).
Apr 24 2012
next sibling parent reply Brad Roberts <braddr slice-2.puremagic.com> writes:
On Tue, 24 Apr 2012, Eljay wrote:

 As a follow up to my email to Walter...
 
 I know I didn't address the question "How can D become adopted at my company?"
 head-on.

Your response is actually very typical of most responses to the question. What's interesting to me is that it's really a deflection and dodges the entire point of the question. By avoiding the question, you (and don't take this personally, I mean 'the person answering a different question') avoid committing to trying to find a way at all.
 An on-going project written in (say) C++ is not going to get approval to
 re-write in D.  There is no ROI in it.

Neither Walter (in this case) nor the question asked for re-writting anything. In fact, that's frequently stated (again, by Walter and others, including myself) as explicitly a non-goal. Rewriting applications to another language is an exercise in time wasting and bug-reintroduction. Unless you have _another_ driving reason to do a rewrite, don't.
 A new project that could be written in D will be met with a lot of resistance.
 Management will consider D too risky, as compared to writing the same project
 in C++ or C# or Java.  Co-workers not familiar with D will consider it as a
 pain-in-the-learning-curve [an attitude I cannot fathom; learning a new
 computer language is a joy, like opening a birthday present].

And this is finally getting a the heart of the question, but also approaching it with an intend to fail approach to it. Of course you don't want to take something new and introduce it as the solution for the next huge risky project. That's bound to be smacked down and get no where. To introduce change and reduce risk, you start small. Something that's safe to let fail. Of course that can backfire too if you want it to: "See, it failed, so the tools we used must suck." Except that might not actually be why it failed. So, the obvious follow up.. what have I done with D where I work? Little, other than get it on the approved list of software we can use. It's not on the list of officially supported languages (more a defacto thing than an actual list). But the key problem is that I haven't written any new code in a very long time, something I miss more and more. The applications I do touch are all pre-existing code bases, so see above about rewriting. My 2 cents, Brad
Apr 24 2012
parent mta`chrono <chrono mta-international.net> writes:
Am 24.04.2012 21:53, schrieb Brad Roberts:
 Neither Walter (in this case) nor the question asked for re-writting 
 anything.  In fact, that's frequently stated (again, by Walter and others, 
 including myself) as explicitly a non-goal.  Rewriting applications to 
 another language is an exercise in time wasting and bug-reintroduction.  
 Unless you have _another_ driving reason to do a rewrite, don't.
 
 So, the obvious follow up.. what have I done with D where I work?  Little, 
 other than get it on the approved list of software we can use.  It's not 
 on the list of officially supported languages (more a defacto thing than 
 an actual list).  But the key problem is that I haven't written any new 
 code in a very long time, something I miss more and more.  The 
 applications I do touch are all pre-existing code bases, so see above 
 about rewriting.
 
 My 2 cents,
 Brad
 

Exactly!!! That's the point. I fully agree with this and we should take account to this in every furthur endeavours. D must be seamlessly integratable with any kind of existing codebase.
Apr 24 2012
prev sibling next sibling parent "Nick Sabalausky" <SeeWebsiteToContactMe semitwist.com> writes:
"SomeDude" <lovelydear mailmetrash.com> wrote in message 
news:zmlqmuhznaynwtcyplof forum.dlang.org...
 On Saturday, 28 April 2012 at 09:12:23 UTC, SomeDude wrote:
 Real time guarantees on a GC is not something we are going to offer 
 anytime soon anyway. While a minimal library, loosely based on the C 
 standard library, with some more bells and whistles that could be 
 borrowed from Phobos, this is a goal that is achievable in a foreseeable 
 future. And both game developers and embedded programmers would be 
 interested.

Note that Kenta Cho, who wrote fast games in D1,

Actually, I think it was pre-D1. (They were fantastic games, too.)
 used this approach, and it worked very well for him.

Interesting, I had wondered about that. I never dug quite that deep into the code, so I never knew he had done it that way.
Apr 29 2012
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/29/2012 2:38 AM, Manu wrote:
 I'm interested to know is whether using a new precise GC will guarantee ALL
 unreferenced stuff will be cleaned on any given sweep.

The new hook put into the typeinfo will do precise collection for references within GC allocated objects. For references that sit on the stack or in static data, it will still use the current conservative scheme. Some things will always be imprecise, like if you have a union of a pointer with an integer, or if you allocate untyped data.
Apr 29 2012
prev sibling parent Don Clugston <dac nospam.com> writes:
On 30/04/12 01:03, Manu wrote:
 On 30 April 2012 01:24, Tove <tove fransson.se
 <mailto:tove fransson.se>> wrote:

     On Sunday, 29 April 2012 at 22:13:22 UTC, Manu wrote:

         Is it technically possible to have a precise GC clean up all
         unreferenced
         memory in one big pass?


     yes, but unless it's also moving/compacting... one would suffer
     memory fragmentation... so I would imagine TempAlloc is a better fit?


 In some cases I'm comfortable with that type of fragmentation (large
 regularly sized resources), although that leads me to a gaping hole in
 D's allocation system...

 <OT, but still very important>
 There is no way to request aligned memory. I can't even specify an
 alignment on a user type and expect it to be aligned if I create one on
 the stack, let alone the heap >_<
 It seems I can request alignment for items within a struct, but I can't
 align the struct its self. In addition, a struct doesn't inherit the
 alignment of its aligned members, so the struct is allocated unaligned,
 and the aligned member fails its promise anyway.

Bug 2278.
May 03 2012
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Tuesday, 24 April 2012 at 12:04:27 UTC, Eljay wrote:
  I have to say, I’m not a fan of JavaScript.  I’ve seen the 
 leading edge of compile-to-JavaScript languages, such as 
 CoffeeScript and DART.  Can D get on that bandwagon and have 
 the D compiler compile to JavaScript as some sort of IL?

Yeah, I've had more success than I thought I would with a dmd fork: https://github.com/adamdruppe/dmd/tree/dtojs A good chunk of the language works and we can do pretty good library stuff (when I find the time!) there's also LLVM's emscripten that takes a different approach.
Apr 24 2012
prev sibling next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Eljay:

 Looking at all the successful languages, I have noticed that 
 all the successful ones I am familiar with have had some sort 
 of sponsor pushing the technology.

Python was widely used before Google "support". And I think Haskell has enjoyed corporate support for a lot of time.
 My understanding is that Facebook is sponsoring D.

Not much, I think.
 *** What does D lack? ***

Sometimes the problem is having too much ;-)
 But the final result would showcase that D can do the heavy 
 lifting of an operating system.  Let’s see /language-X/ do 
 that!  (Where /language-X/ is not Assembly, C, C++, or Sing#.)

I think Walter doesn't believe a lot in Sing#... Bye, bearophile
Apr 24 2012
next sibling parent =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <xtzgzorex gmail.com> writes:
On 24-04-2012 16:05, bearophile wrote:
 Eljay:

 Looking at all the successful languages, I have noticed that all the
 successful ones I am familiar with have had some sort of sponsor
 pushing the technology.

Python was widely used before Google "support". And I think Haskell has enjoyed corporate support for a lot of time.
 My understanding is that Facebook is sponsoring D.

Not much, I think.
 *** What does D lack? ***

Sometimes the problem is having too much ;-)
 But the final result would showcase that D can do the heavy lifting of
 an operating system. Let’s see /language-X/ do that! (Where
 /language-X/ is not Assembly, C, C++, or Sing#.)

I think Walter doesn't believe a lot in Sing#... Bye, bearophile

Sing# has been battle-tested to build an actual operating system. I don't think it's a matter of belief... -- - Alex
Apr 24 2012
prev sibling parent reply Don Clugston <dac nospam.com> writes:
On 25/04/12 17:38, Joseph Rushton Wakeling wrote:
 On 25/04/12 16:58, Kagamin wrote:
 On Tuesday, 24 April 2012 at 14:05:14 UTC, bearophile wrote:
 Python was widely used before Google "support". And I think Haskell has
 enjoyed corporate support for a lot of time.

And who's behind PHP?

... but importantly, Python and PHP (and Ruby, and Haskell, and others) were fully open source in their reference implementations from the get-go, or at least from very early on. This isn't just important in itself, but has a multiplicative impact with inclusion in the Linux distros, BSD's, etc. which make up the server infrastructure of the web. It also enables all sorts of 3rd-party suppliers who feel comfortable including the software in their hosting provision because they can be certain they won't in future suffer from the commercial constraints of a proprietary supplier. D's reference implementation _still_ isn't fully open source -- only the frontend -- and the available open source compilers lag behind the reference.

<rant> "open source" is a horrible, duplicitous term. Really what you mean is "the license is not GPL compatible". </rant> Based on my understanding of the legal situation with Symantec, the backend CANNOT become GPL compatible. Stop using the word "still", it will NEVER happen.
Apr 26 2012
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-04-26 11:07, Don Clugston wrote:

 Based on my understanding of the legal situation with Symantec, the
 backend CANNOT become GPL compatible. Stop using the word "still", it
 will NEVER happen.

Theoretically someone could: A. Replace all parts of the backend that Symantec can't/won't license as GPL (don't know if that is the whole backend or not) B. Buy the backend from Symantec -- /Jacob Carlborg
Apr 26 2012
prev sibling next sibling parent reply Don Clugston <dac nospam.com> writes:
On 26/04/12 14:58, Joseph Rushton Wakeling wrote:
 On 26/04/12 11:07, Don Clugston wrote:
 <rant>
 "open source" is a horrible, duplicitous term. Really what you mean is
 "the
 license is not GPL compatible".
 </rant>

No, I don't mean "GPL compatible". I'd be perfectly happy for the DMD backend to be released under a GPL-incompatible free/open source licence like the CDDL. The problem is not GPL compatibility but whether sufficient freedoms are granted to distribute and modify sources.

And the only one such limitation of freedom which has ever been identified, in numerous posts (hundreds!) on this topic, is that the license is not GPL compatible and therefore cannot be distributed with (say) OS distributions. Everything else is FUD.
Apr 26 2012
parent Jeff Nowakowski <jeff dilacero.org> writes:
On 04/26/2012 10:59 AM, Don Clugston wrote:
 No, I don't mean "GPL compatible". I'd be perfectly happy for the DMD
 backend to be released under a GPL-incompatible free/open source licence
 like the CDDL.

 The problem is not GPL compatibility but whether sufficient freedoms are
 granted to distribute and modify sources.

And the only one such limitation of freedom which has ever been identified, in numerous posts (hundreds!) on this topic, is that the license is not GPL compatible and therefore cannot be distributed with (say) OS distributions.

I don't understand your fixation on the GPL, as even a GPL-incompatible license would allow it to be distributed on FOSS operating systems like Debian or Fedora. The important principle, which you've been ignoring for some reason, is that you can redistribute the source along with modifications. This is not special to GPL, and is fundamental both to open source and Free Software.
Apr 27 2012
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/26/2012 2:27 AM, Jonathan M Davis wrote:
 I think that the "openness" of dmd being an issue is purely  a matter of
 misunderstandings and FUD. And if Walter _could_ make the backend GPL, he may
 very well have done so ages ago. But he can't, so there's no point in
 complaining about it - especially since it doesn't impede your ability to use
 dmd.

I have tried, but failed. I also agree with you that it's moot, as LDC and GDC exist.
Apr 26 2012
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Tuesday, 24 April 2012 at 14:05:14 UTC, bearophile wrote:
 Eljay:

 Looking at all the successful languages, I have noticed that 
 all the successful ones I am familiar with have had some sort 
 of sponsor pushing the technology.

Python was widely used before Google "support". And I think Haskell has enjoyed corporate support for a lot of time.

Python's killer application was Zope. I recall before Zope, no one cared about Python in Portugal, only afterwards, people started taking Python seriously, Some of the main Haskell researchers are in the payroll of companies like Microsoft, or Siemens, for example. The proprietary languages usually are pushed by big companies, until you cannot avoid them. While the, lets call them, community oriented languages, really need something that makes people care for the language and introduce them silently in the company. I played a bit with D1, but never cared about it too much. What really made me give a second look to it was Andrei's book, but then I was disappointed to find out that not everything was really working as described in the book. As a language geek, I toy around with all programming languages I can play with, but I see the same issues as raised by Eljay.
Apr 24 2012
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-04-24 14:04, Eljay wrote:

 D on iOS. So for me personally, I would love to use D to write my
 applications for iOS, and OS X. But… I’m not sure how to do that. (The
 Objective-D project looks abandoned, never got out of the “toy” project
 stage, and doesn’t bridge Cocoa’s Frameworks written in Objective-C to
 D/Objective-D anyway.)

You would need to write bindings to the Objective-C classes just as you need to write bindings to the C functions you want to use. I'm currently working on a tool that does this automatically. As a first step I intend to support C and Objective-C, then probably C++ as well. -- /Jacob Carlborg
Apr 24 2012
prev sibling next sibling parent "Kagamin" <spam here.lot> writes:
On Tuesday, 24 April 2012 at 14:05:14 UTC, bearophile wrote:
 Eljay:

 Looking at all the successful languages, I have noticed that 
 all the successful ones I am familiar with have had some sort 
 of sponsor pushing the technology.

Python was widely used before Google "support". And I think Haskell has enjoyed corporate support for a lot of time.

And who's behind PHP?
Apr 25 2012
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Wednesday, 25 April 2012 at 14:58:13 UTC, Kagamin wrote:
 On Tuesday, 24 April 2012 at 14:05:14 UTC, bearophile wrote:
 Eljay:

 Looking at all the successful languages, I have noticed that 
 all the successful ones I am familiar with have had some sort 
 of sponsor pushing the technology.

Python was widely used before Google "support". And I think Haskell has enjoyed corporate support for a lot of time.

And who's behind PHP?

Zend + the endless amount of ISPs that offer only cheap PHP installations, while charging endless amount of money for other types of server deployments.
Apr 25 2012
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 25/04/12 16:58, Kagamin wrote:
 On Tuesday, 24 April 2012 at 14:05:14 UTC, bearophile wrote:
 Python was widely used before Google "support". And I think Haskell has
 enjoyed corporate support for a lot of time.

And who's behind PHP?

... but importantly, Python and PHP (and Ruby, and Haskell, and others) were fully open source in their reference implementations from the get-go, or at least from very early on. This isn't just important in itself, but has a multiplicative impact with inclusion in the Linux distros, BSD's, etc. which make up the server infrastructure of the web. It also enables all sorts of 3rd-party suppliers who feel comfortable including the software in their hosting provision because they can be certain they won't in future suffer from the commercial constraints of a proprietary supplier. D's reference implementation _still_ isn't fully open source -- only the frontend -- and the available open source compilers lag behind the reference.
Apr 25 2012
prev sibling next sibling parent "Eljay" <eljay451 gmail.com> writes:
Thank you Brad, that's the kind of response I was hoping to 
elicit.

 What's interesting to me is that it's really a deflection and 
 dodges the entire point of the question.

Yes, I know. I tried to step back and look at the bigger picture, and the issue of "what are the pain points which hinder D from being used". As well as "what could the D community do to make D a more compelling alternative". Even when I had my own one-man company, and could use any programming language I wanted -- and despite my own unbridled enthusiasm for D -- I ended up not using D.
Apr 25 2012
prev sibling next sibling parent Brad Roberts <braddr puremagic.com> writes:
On 4/25/2012 1:37 PM, Eljay wrote:
 Thank you Brad, that's the kind of response I was hoping to elicit.
 
 What's interesting to me is that it's really a deflection and dodges the
entire point of the question.

Yes, I know. I tried to step back and look at the bigger picture, and the issue of "what are the pain points which hinder D from being used". As well as "what could the D community do to make D a more compelling alternative". Even when I had my own one-man company, and could use any programming language I wanted -- and despite my own unbridled enthusiasm for D -- I ended up not using D.

Part of my point is that it's _easy_ to find reasons to not introduce change, regardless of the nature of the change. Even if the change is something that's low risk and done or used all the time. It takes a little bravery and faith and determination to cause change. It takes even more to make risky changes, and no doubt, using D carries risks. BUT, unless those risks are taken, the status quo won't change. It's a lot like interviewing potential employees. It's really pretty easy to seek out reasons not to hire and pass on every candidate. I know people that take that approach with their interviews.. and quickly get taken aside and re-trained how to interview or are just removed from the process altogether. It takes a balanced approach. We don't need more generalizations about why not to use D, we need people willing to take a minor risk and introduce D to demonstrate its strengths and accept the warts knowing that the trend is clearly in the right direction. Another 2 cents, Brad
Apr 25 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, April 26, 2012 11:07:04 Don Clugston wrote:
 <rant>
 "open source" is a horrible, duplicitous term. Really what you mean is
 "the license is not GPL compatible".
 </rant>
 
 Based on my understanding of the legal situation with Symantec, the
 backend CANNOT become GPL compatible. Stop using the word "still", it
 will NEVER happen.

And it really doesn't need to. I honestly don't understand why it's an issue at all other than people completely misunderstanding the situation or being the types of folks who think that anything which isn't completely and totally open is evil. Whether the backend is open or not has _zero_ impact on your ability to use it. The source is freely available, so you can look at and see what it does. You can even submit pull requests for it. Yes, there are some limitations on you going and doing whatever you want with the source, but so what? There's _nothing_ impeding your ability to use it to compile programs. And the front- end - which is really where D itself is - _is_ under the GPL. Not to mention, if really want a "fully open" D compiler, there's always gdc and ldc, so you there _are_ alternatives. The fact that dmd isn't really doesn't affect much except for the people whom are overzealous about "free software." I think that the "openness" of dmd being an issue is purely a matter of misunderstandings and FUD. And if Walter _could_ make the backend GPL, he may very well have done so ages ago. But he can't, so there's no point in complaining about it - especially since it doesn't impede your ability to use dmd. - Jonathan M Davis
Apr 26 2012
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 26/04/12 11:07, Don Clugston wrote:
 <rant>
 "open source" is a horrible, duplicitous term. Really what you mean is "the
 license is not GPL compatible".
 </rant>

No, I don't mean "GPL compatible". I'd be perfectly happy for the DMD backend to be released under a GPL-incompatible free/open source licence like the CDDL. The problem is not GPL compatibility but whether sufficient freedoms are granted to distribute and modify sources. That has a knockon impact on the ability of 3rd parties to package and distribute the software, to patch it without necessarily going via upstream, etc. etc., all of which affects the degree to which others can easily use the language.
 Based on my understanding of the legal situation with Symantec, the backend
 CANNOT become GPL compatible. Stop using the word "still", it will NEVER
happen.

Please understand that I'm not suggesting any bad faith on the part of D's developers. Walter's good intentions are clear in the strong support he's given to GDC and other freely-licensed compilers. All I'm suggesting is that being free software (a somewhat better-defined term) was a key factor in some languages gaining popularity without corporate backing, and that the non-free nature of the DMD backend may have prevented D from enjoying this potential source of support. On 26/04/12 11:27, Jonathan M Davis wrote:
 And it really doesn't need to. I honestly don't understand why it's an issue
 at all other than people completely misunderstanding the situation or being
 the types of folks who think that anything which isn't completely and totally
 open is evil.

 Whether the backend is open or not has _zero_ impact on your ability to use
 it. The source is freely available, so you can look at and see what it does.
 You can even submit pull requests for it. Yes, there are some limitations on
 you going and doing  whatever you want with the source, but so what? There's
 _nothing_ impeding your ability to use it to compile programs. And the front-
 end - which is really where D itself is - _is_ under the GPL.

You misunderstand my point. I'm not saying anyone is evil; I'm simply pointing out that the licensing constraints prevent various kinds of 3rd party distribution and engagement that could be useful in spreading awareness and use of the language. That _does_ have an impact on use, in terms of constraining the development of 3rd-party support and infrastructure.
 Not to mention, if really want a "fully open" D compiler, there's always gdc
 and ldc, so you there _are_ alternatives. The fact that dmd isn't really
 doesn't affect much except for the people whom are overzealous about "free
 software."

Yes, but GDC and LDC both (for now) lag behind DMD in terms of functionality -- I was not able to compile my updates to Phobos using GDC -- and it's almost inevitable that they will always have to play catch-up, even though the impact of that will lessen over time. That's why I spoke about the "reference implementation" of the language: D2 has been available for quite some time now, but it's only last Autumn that a D2 compiler landed in my Linux distro.
 I think that the "openness" of dmd being an issue is purely  a matter of
 misunderstandings and FUD. And if Walter _could_ make the backend GPL, he may
 very well have done so ages ago. But he can't, so there's no point in
 complaining about it - especially since it doesn't impede your ability to use
 dmd.

To an extent I agree with you. The good intentions of Walter and the other D developers are clear, it's always been apparent that there will be fully open source compilers for the language, etc. etc.; I wouldn't be here if I wasn't happy to work with DMD under its given licence terms. But it's not FUD to say that the licensing does make more difficult certain kinds of engagement that have been very helpful for other languages, such as inclusion in Linux distros and BSD's or other software collections -- and that has a further impact in those suppliers' willingness or ability to ship other software written in D. It's also fair to say that if the licensing was different, that would remove an entire source of potential FUD. Again, I'm not saying that anyone is evil, that I find the situation personally unacceptable or that I don't understand the reasons why things are as they are. I just made the point that _being_ free/open source software was probably an important factor in the success of a number of now-popular languages that didn't originally enjoy corporate support, and that the licensing of the DMD backend prevents it from enjoying some of those avenues to success. .... and I _want_ to see that success, because I think D deserves it. Best wishes, -- Joe
Apr 26 2012
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 26/04/12 16:59, Don Clugston wrote:
 And the only one such limitation of freedom which has ever been identified, in
 numerous posts (hundreds!) on this topic, is that the license is not GPL
 compatible and therefore cannot be distributed with (say) OS distributions.

Yes, I appreciate I touched on a sore point and one that must have been discussed to death. I wasn't meaning to add to the noise, but your response to my original email was so hostile I felt I had to reply at length to clarify. I personally don't think it's a minor issue that the reference version of D can't be included with open source distributions, but I also think there are much more pressing immediate issues than this to resolve in the short term. By the way, there are plenty of non-GPL-compatible licences that have traditionally been considered acceptable by open source distributions -- the original Mozilla Public Licence and Apache Licence (new versions have since been released which ensure compatibility), at least one variant of the permissive BSD/MIT licences, and probably others. It's whether the licence implements the "four freedoms" that matters.
Apr 26 2012
prev sibling next sibling parent Sean Kelly <sean invisibleduck.org> writes:
On Apr 26, 2012, at 5:58 AM, Joseph Rushton Wakeling wrote:

 The problem is not GPL compatibility but whether sufficient freedoms =

on the ability of 3rd parties to package and distribute the software, to = patch it without necessarily going via upstream, etc. etc., all of which = affects the degree to which others can easily use the language. While distributing modified sources is certainly one way of dealing with = changes not represented by the official distribution, I prefer = distributing patches instead. It's easier to audit what's being = changed, and updating to a new release tends to be easier.=
Apr 26 2012
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 26/04/12 21:08, Walter Bright wrote:
 I have tried, but failed.

 I also agree with you that it's moot, as LDC and GDC exist.

I think I should probably add here that I do recognize the amount of effort you've put in here, and wasn't intending to be pejorative about DMD. I just think it's a terrible shame that you've been constrained in this way.
Apr 26 2012
prev sibling next sibling parent "Kagamin" <spam here.lot> writes:
On Thursday, 26 April 2012 at 09:28:30 UTC, Jonathan M Davis 
wrote:
 Whether the backend is open or not has _zero_ impact on your 
 ability to use
 it. The source is freely available, so you can look at and see 
 what it does.

Casual users are generally ignorant about licenses (as long as they can use the software), but not geeks - and proprietary software has bad publicity, it's not something technical, just a reputation.
Apr 27 2012
prev sibling next sibling parent "SomeDude" <lovelydear mailmetrash.com> writes:
On Tuesday, 24 April 2012 at 12:50:27 UTC, Eljay wrote:
 ---

 And the most important bit of information:  I use vi (Vim).

I think that the type of application where D is proving itself right now is high performance server applications, and particularly web servers. D seems completely fit to replace Java on most server apps, with both better performance and better memory usage. The web interface to the newsgroups, as well as the recently revealed vibe.d web server seem to support this view. D can handle both batch and real time treatments really well I think. That is where it can gain a lot of weight in the enterprise, even before games and scientific applications.
Apr 27 2012
prev sibling next sibling parent "SomeDude" <lovelydear mailmetrash.com> writes:
On Friday, 27 April 2012 at 23:28:09 UTC, SomeDude wrote:
 On Tuesday, 24 April 2012 at 12:50:27 UTC, Eljay wrote:
 ---

 And the most important bit of information:  I use vi (Vim).

I think that the type of application where D is proving itself right now is high performance server applications, and particularly web servers. D seems completely fit to replace Java on most server apps, with both better performance and better memory usage. The web interface to the newsgroups, as well as the recently revealed vibe.d web server seem to support this view. D can handle both batch and real time treatments really well I think. That is where it can gain a lot of weight in the enterprise, even before games and scientific applications.

The other thing that would make it attractive among the C++ developers, would be the development of a lightweight, high performance, minimal library that doesn't use the GC at all. Ideally, it would be compatible with Phobos. I bet if such a library existed, flocks of C++ developers would suddenly switch to D.
Apr 27 2012
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Apr 28, 2012 at 01:31:32AM +0200, SomeDude wrote:
[...]
 The other thing that would make it attractive among the C++
 developers, would be the development of a lightweight, high
 performance, minimal library that doesn't use the GC at all.  Ideally,
 it would be compatible with Phobos. I bet if such a library existed,
 flocks of C++ developers would suddenly switch to D.

I know the current GC leaves much room for improvement, but what's the hangup about the GC anyway? If -- and yes this is a very big if -- the GC has real-time guarantees, would that make it more palatable to C++ devs? Or is it just because they have trouble with the idea of having a GC in the first place? T -- Three out of two people have difficulties with fractions. -- Dirk Eddelbuettel
Apr 27 2012
prev sibling next sibling parent "SomeDude" <lovelydear mailmetrash.com> writes:
On Saturday, 28 April 2012 at 01:09:25 UTC, H. S. Teoh wrote:
 On Sat, Apr 28, 2012 at 01:31:32AM +0200, SomeDude wrote:
 [...]
 The other thing that would make it attractive among the C++
 developers, would be the development of a lightweight, high
 performance, minimal library that doesn't use the GC at all.  
 Ideally,
 it would be compatible with Phobos. I bet if such a library 
 existed,
 flocks of C++ developers would suddenly switch to D.

I know the current GC leaves much room for improvement, but what's the hangup about the GC anyway? If -- and yes this is a very big if -- the GC has real-time guarantees, would that make it more palatable to C++ devs? Or is it just because they have trouble with the idea of having a GC in the first place? T

Real time guarantees on a GC is not something we are going to offer anytime soon anyway. While a minimal library, loosely based on the C standard library, with some more bells and whistles that could be borrowed from Phobos, this is a goal that is achievable in a foreseeable future. And both game developers and embedded programmers would be interested.
Apr 28 2012
prev sibling next sibling parent "SomeDude" <lovelydear mailmetrash.com> writes:
On Saturday, 28 April 2012 at 09:12:23 UTC, SomeDude wrote:
 Real time guarantees on a GC is not something we are going to 
 offer anytime soon anyway. While a minimal library, loosely 
 based on the C standard library, with some more bells and 
 whistles that could be borrowed from Phobos, this is a goal 
 that is achievable in a foreseeable future. And both game 
 developers and embedded programmers would be interested.

Note that Kenta Cho, who wrote fast games in D1, used this approach, and it worked very well for him.
Apr 28 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, April 28, 2012 11:12:21 SomeDude wrote:
 Real time guarantees on a GC is not something we are going to
 offer anytime soon anyway. While a minimal library, loosely based
 on the C standard library, with some more bells and whistles that
 could be borrowed from Phobos, this is a goal that is achievable
 in a foreseeable future. And both game developers and embedded
 programmers would be interested.

If what you want is the C standard library, then use the C standard library. There's nothing stopping you, and trying to replicate it in D would be pointless. The main problems with the GC in Phobos are likely arrays and containers. You can't fix the array problem. If you do _anything_ which involves slicing or any array functions which could allocate, you're going to need the GC. The only way to avoid the problem completely is to restrict the functions that you use with arrays to those which won't append to an array or otherwise allocate memory for an array. The container problem should be resolved via custom allocators once they've been added. The custom allocators will also help reduce GC issues for classes in general. But in general, by minimizing how much you do which would require the GC, the little that does shouldn't be a big deal. Still, due to how arrays work, there's really no way to get away from the GC completely without restricting what you do with them, which in some cases means not using Phobos. I don't think that there's really any way around that. - Jonathan M Davis
Apr 28 2012
prev sibling next sibling parent "SomeDude" <lovelydear mailmetrash.com> writes:
On Saturday, 28 April 2012 at 09:22:35 UTC, Jonathan M Davis 
wrote:
 On Saturday, April 28, 2012 11:12:21 SomeDude wrote:
 Real time guarantees on a GC is not something we are going to
 offer anytime soon anyway. While a minimal library, loosely 
 based
 on the C standard library, with some more bells and whistles 
 that
 could be borrowed from Phobos, this is a goal that is 
 achievable
 in a foreseeable future. And both game developers and embedded
 programmers would be interested.

If what you want is the C standard library, then use the C standard library. There's nothing stopping you, and trying to replicate it in D would be pointless. The main problems with the GC in Phobos are likely arrays and containers. You can't fix the array problem. If you do _anything_ which involves slicing or any array functions which could allocate, you're going to need the GC. The only way to avoid the problem completely is to restrict the functions that you use with arrays to those which won't append to an array or otherwise allocate memory for an array. The container problem should be resolved via custom allocators once they've been added. The custom allocators will also help reduce GC issues for classes in general. But in general, by minimizing how much you do which would require the GC, the little that does shouldn't be a big deal. Still, due to how arrays work, there's really no way to get away from the GC completely without restricting what you do with them, which in some cases means not using Phobos. I don't think that there's really any way around that. - Jonathan M Davis

Right, I understand the situation better now. So basically, what's needed is the custom allocators, and the GC would be relieved from much of the work. That would still not work for hard real time embedded, but for those applications, there are lots of restrictions on memory anyway (no dynamic allocation for once), so it wouldn't change much.
Apr 28 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, April 28, 2012 11:35:19 SomeDude wrote:
 Right, I understand the situation better now. So basically,
 what's needed is the custom allocators, and the GC would be
 relieved from much of the work. That would still not work for
 hard real time embedded, but for those applications, there are
 lots of restrictions on memory anyway (no dynamic allocation for
 once), so it wouldn't change much.

With custom allocators and/or shared pointers/references, you can pretty much avoid the GC entirely for classes as well as any structs that you put on the heap. So, you'd be in essentially the same place that C++ is for that. It's just arrays that you can't really fix. If you restrict yourself to what C/C++ can do with arrays (plus taking advantage of the length property), then you're fine, but if you do much beyond that, then you need the GC or you're going to have problems. So, as long as you're careful with arrays, you should be able to have the memory situation be pretty much identical to what it is in C/C++. And, of course, if you can afford to use the GC in at least some of your code, then it's there to use. I believe that the typical approach however is to use the GC unless profiling indicates that it's causing you performance problems somewhere, and then you optimize that code so that it minimizes its GC usage or so that it avoids the GC entirely. That way, your program as a whole can reap the benefits granted by the GC, but your performance-critical code can still be performant. Actually, now that I think about it, delegates would be another area where you'd have to be careful, since they generally end up having to have closures allocated for them when you pass them to a function unless that function them takes them as scope parameters. But it's easy to avoid using delegates if you want to. And if you want to program in a subset of the language that's closer to C, then you probably wouldn't be using them anyway. - Jonathan M Davis
Apr 28 2012
prev sibling next sibling parent "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Saturday, 28 April 2012 at 09:14:51 UTC, SomeDude wrote:
 On Saturday, 28 April 2012 at 09:12:23 UTC, SomeDude wrote:
 Real time guarantees on a GC is not something we are going to 
 offer anytime soon anyway. While a minimal library, loosely 
 based on the C standard library, with some more bells and 
 whistles that could be borrowed from Phobos, this is a goal 
 that is achievable in a foreseeable future. And both game 
 developers and embedded programmers would be interested.

Note that Kenta Cho, who wrote fast games in D1, used this approach, and it worked very well for him.

I also write games in D. My approach is this: use the GC all you want during loading or other non-interactive parts of the game and then just make sure that you don't use it during gameplay. GC vs. manual memory allocation is a non-issue for real-time guarantees. The simple fact of the matter is that you should be using neither. I also don't use malloc/free during runtime because it has the same non-real-time problems as using the GC. A single malloc can stall for tens of milliseconds or more, and that's simply too much. Just learn how to write code that doesn't allocate memory. A bigger problem with GC for games is memory management i.e. controlling how much memory is currently allocated, and what systems are using what memory. Having deterministic memory usage is preferable for those cases because I know that as soon as I delete something that the memory is available for something else. I don't get that guarantee with a GC.
Apr 28 2012
prev sibling next sibling parent reply Manu <turkeyman gmail.com> writes:
--20cf3074b41ab42d2e04becde03a
Content-Type: text/plain; charset=UTF-8

On 28 April 2012 04:10, H. S. Teoh <hsteoh quickfur.ath.cx> wrote:

 On Sat, Apr 28, 2012 at 01:31:32AM +0200, SomeDude wrote:
 [...]
 The other thing that would make it attractive among the C++
 developers, would be the development of a lightweight, high
 performance, minimal library that doesn't use the GC at all.  Ideally,
 it would be compatible with Phobos. I bet if such a library existed,
 flocks of C++ developers would suddenly switch to D.

I know the current GC leaves much room for improvement, but what's the hangup about the GC anyway? If -- and yes this is a very big if -- the GC has real-time guarantees, would that make it more palatable to C++ devs? Or is it just because they have trouble with the idea of having a GC in the first place?

If the GC guarantees to behave in a deterministic and predictable way, I have no problem with it. And even if it doesn't, as long as it's lightning fast, and I can control the sweeps. One major concern to me is invisible allocations. I want to know when I'm allocating, I like allocate operations to be clearly visible. There are a lot of operations that cause invisible allocations in D, but they are avoidable. Games are both embedded and realtime code at the same time, this unions the strict requirements of both worlds into a system that demands very tight control of these things. Fragmentation is the enemy, so is losing 1ms (GC takes WAY longer than this currently) at random moments. There is a problem right now where the GC doesn't actually seem to work, and I'm seeing D apps allocating gigabytes and never releasing the memory. A great case study for the GC is VisualD, if any GC experts would like to check it out. It shows a use case where the GC utterly fails, and makes the software borderline unusable as a result. It seems to 'leak' memory, and collects can take 5-10 seconds at a time (manifested by locking up the entire application). VisualD has completely undermined by faith and trust in the GC, and I've basically banned using it. I can't afford to run into that situation a few months down the line. --20cf3074b41ab42d2e04becde03a Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div class=3D"gmail_extra"><div class=3D"gmail_quote">On 28 April 2012 04:1= 0, H. S. Teoh <span dir=3D"ltr">&lt;<a href=3D"mailto:hsteoh quickfur.ath.c= x" target=3D"_blank">hsteoh quickfur.ath.cx</a>&gt;</span> wrote:<br><block= quote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc= solid;padding-left:1ex"> On Sat, Apr 28, 2012 at 01:31:32AM +0200, SomeDude wrote:<br> [...]<br> <div class=3D"im">&gt; The other thing that would make it attractive among = the C++<br> &gt; developers, would be the development of a lightweight, high<br> &gt; performance, minimal library that doesn&#39;t use the GC at all. =C2= =A0Ideally,<br> &gt; it would be compatible with Phobos. I bet if such a library existed,<b= r> &gt; flocks of C++ developers would suddenly switch to D.<br> <br> </div>I know the current GC leaves much room for improvement, but what&#39;= s the<br> hangup about the GC anyway? If -- and yes this is a very big if -- the<br> GC has real-time guarantees, would that make it more palatable to C++<br> devs? Or is it just because they have trouble with the idea of having a<br> GC in the first place?<br></blockquote><div><br></div><div>If the GC guaran= tees to behave in a deterministic and predictable way, I have no problem wi= th it. And even if it doesn&#39;t, as long as it&#39;s lightning fast, and = I can control the sweeps.</div> <div>One major concern to me is invisible allocations. I want to know when = I&#39;m allocating, I like allocate operations to be clearly visible. There= are a lot of operations that cause invisible allocations in D, but they ar= e avoidable.</div> <div>Games are both embedded and realtime code at the same time, this union= s the strict requirements of both worlds into a system that demands very ti= ght control of these things. Fragmentation is the enemy, so is losing 1ms (= GC takes WAY longer than this currently) at random moments.</div> <div><br></div><div>There is a problem right now where the GC doesn&#39;t a= ctually seem to work, and I&#39;m seeing D apps allocating gigabytes and ne= ver releasing the memory.</div><div>A great case study for the GC is Visual= D, if any GC experts would like to check it out. It shows a use case where = the GC utterly fails, and makes the software borderline unusable as a resul= t. It seems to &#39;leak&#39; memory, and collects can take 5-10 seconds at= a time (manifested by locking up the entire application).</div> <div>VisualD has completely undermined by faith and trust in the GC, and I&= #39;ve basically banned using it. I can&#39;t afford to run into that situa= tion a few months down the line.</div></div></div> --20cf3074b41ab42d2e04becde03a--
Apr 29 2012
parent reply "Nick Sabalausky" <SeeWebsiteToContactMe semitwist.com> writes:
"Manu" <turkeyman gmail.com> wrote in message 
news:mailman.93.1335691450.24740.digitalmars-d puremagic.com...
 On 28 April 2012 04:10, H. S. Teoh <hsteoh quickfur.ath.cx> wrote:

 On Sat, Apr 28, 2012 at 01:31:32AM +0200, SomeDude wrote:
 [...]
 The other thing that would make it attractive among the C++
 developers, would be the development of a lightweight, high
 performance, minimal library that doesn't use the GC at all.  Ideally,
 it would be compatible with Phobos. I bet if such a library existed,
 flocks of C++ developers would suddenly switch to D.

I know the current GC leaves much room for improvement, but what's the hangup about the GC anyway? If -- and yes this is a very big if -- the GC has real-time guarantees, would that make it more palatable to C++ devs? Or is it just because they have trouble with the idea of having a GC in the first place?

If the GC guarantees to behave in a deterministic and predictable way, I have no problem with it. And even if it doesn't, as long as it's lightning fast, and I can control the sweeps. One major concern to me is invisible allocations. I want to know when I'm allocating, I like allocate operations to be clearly visible. There are a lot of operations that cause invisible allocations in D, but they are avoidable. Games are both embedded and realtime code at the same time, this unions the strict requirements of both worlds into a system that demands very tight control of these things. Fragmentation is the enemy, so is losing 1ms (GC takes WAY longer than this currently) at random moments. There is a problem right now where the GC doesn't actually seem to work, and I'm seeing D apps allocating gigabytes and never releasing the memory. A great case study for the GC is VisualD, if any GC experts would like to check it out. It shows a use case where the GC utterly fails, and makes the software borderline unusable as a result. It seems to 'leak' memory, and collects can take 5-10 seconds at a time (manifested by locking up the entire application). VisualD has completely undermined by faith and trust in the GC, and I've basically banned using it. I can't afford to run into that situation a few months down the line.

I once tried to create a batch image processing tool in D, and false pointers rendered the whole thing unusable. I've been wary about such things since. This was a number of years ago, though.
Apr 30 2012
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 04/30/2012 08:45 PM, Nick Sabalausky wrote:
 I once tried to create a batch image processing tool in D, and false
 pointers rendered the whole thing unusable. I've been wary about such things
 since.

 This was a number of years ago, though.

False pointers in the image data? Why would that even be scanned?
Apr 30 2012
parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/30/2012 12:09 PM, Timon Gehr wrote:
 On 04/30/2012 08:45 PM, Nick Sabalausky wrote:
 I once tried to create a batch image processing tool in D, and false
 pointers rendered the whole thing unusable. I've been wary about such things
 since.

 This was a number of years ago, though.

False pointers in the image data? Why would that even be scanned?

Such was scanned in early D. Anyhow, a large allocation is more likely to have false pointers into it than a small one. For very large allocations, I would suggest handling them explicitly rather than using the GC.
Apr 30 2012
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--20cf3074b4baf2241c04bece197c
Content-Type: text/plain; charset=UTF-8

On 28 April 2012 18:16, Peter Alexander <peter.alexander.au gmail.com>wrote:

 On Saturday, 28 April 2012 at 09:14:51 UTC, SomeDude wrote:

 On Saturday, 28 April 2012 at 09:12:23 UTC, SomeDude wrote:

 Real time guarantees on a GC is not something we are going to offer
 anytime soon anyway. While a minimal library, loosely based on the C
 standard library, with some more bells and whistles that could be borrowed
 from Phobos, this is a goal that is achievable in a foreseeable future. And
 both game developers and embedded programmers would be interested.

Note that Kenta Cho, who wrote fast games in D1, used this approach, and it worked very well for him.

I also write games in D. My approach is this: use the GC all you want during loading or other non-interactive parts of the game and then just make sure that you don't use it during gameplay. GC vs. manual memory allocation is a non-issue for real-time guarantees. The simple fact of the matter is that you should be using neither. I also don't use malloc/free during runtime because it has the same non-real-time problems as using the GC. A single malloc can stall for tens of milliseconds or more, and that's simply too much. Just learn how to write code that doesn't allocate memory. A bigger problem with GC for games is memory management i.e. controlling how much memory is currently allocated, and what systems are using what memory. Having deterministic memory usage is preferable for those cases because I know that as soon as I delete something that the memory is available for something else. I don't get that guarantee with a GC.

I think that basically sums it up. I'm interested to know is whether using a new precise GC will guarantee ALL unreferenced stuff will be cleaned on any given sweep. I can imagine a model in games where I could: 1 Use the GC to allocate as much as I like during initialisation 2 During runtime you never allocate anyway, so disable the GC (this is when it is important to know about hidden allocations) 3 During some clean-up, first run the logic to de-reference all things that are no longer required 4 Finally, force a precise GC scan, which should guarantee that all no-longer referenced memory would be cleaned up at that time. This would actually be a very convenient working model for games. But it only works if I know everything that was released will definitely be cleaned, otherwise I may not be ale to allocate the next level (games often allocate all memory a machine has within 100k or so). --20cf3074b4baf2241c04bece197c Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div class=3D"gmail_extra"><div class=3D"gmail_quote">On 28 April 2012 18:1= 6, Peter Alexander <span dir=3D"ltr">&lt;<a href=3D"mailto:peter.alexander.= au gmail.com" target=3D"_blank">peter.alexander.au gmail.com</a>&gt;</span>= wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"><div class=3D"HOEnZb"><div class=3D"h5">On S= aturday, 28 April 2012 at 09:14:51 UTC, SomeDude wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> On Saturday, 28 April 2012 at 09:12:23 UTC, SomeDude wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <br> Real time guarantees on a GC is not something we are going to offer anytime= soon anyway. While a minimal library, loosely based on the C standard libr= ary, with some more bells and whistles that could be borrowed from Phobos, = this is a goal that is achievable in a foreseeable future. And both game de= velopers and embedded programmers would be interested.<br> </blockquote> <br> Note that Kenta Cho, who wrote fast games in D1, used this approach, and it= worked very well for him.<br> </blockquote> <br></div></div> I also write games in D.<br> <br> My approach is this: use the GC all you want during loading or other non-in= teractive parts of the game and then just make sure that you don&#39;t use = it during gameplay.<br> <br> GC vs. manual memory allocation is a non-issue for real-time guarantees. Th= e simple fact of the matter is that you should be using neither. I also don= &#39;t use malloc/free during runtime because it has the same non-real-time= problems as using the GC. A single malloc can stall for tens of millisecon= ds or more, and that&#39;s simply too much.<br> <br> Just learn how to write code that doesn&#39;t allocate memory.<br> <br> A bigger problem with GC for games is memory management i.e. controlling ho= w much memory is currently allocated, and what systems are using what memor= y. Having deterministic memory usage is preferable for those cases because = I know that as soon as I delete something that the memory is available for = something else. I don&#39;t get that guarantee with a GC.<br> </blockquote></div><br></div><div class=3D"gmail_extra">I think that basica= lly sums it up.</div><div class=3D"gmail_extra"><br></div><div class=3D"gma= il_extra">I&#39;m interested to know is whether using a new precise GC will= guarantee ALL unreferenced stuff will be cleaned on any given sweep.</div> <div class=3D"gmail_extra">I can imagine a model in games where I could:</d= iv><div class=3D"gmail_extra">=C2=A01 Use the GC to allocate as much as I l= ike during initialisation</div><div class=3D"gmail_extra">=C2=A02 During ru= ntime you never allocate anyway, so disable the GC (this is when it is impo= rtant to know about hidden allocations)</div> <div class=3D"gmail_extra">=C2=A03 During some clean-up, first run the logi= c to de-reference all things that are no longer required</div><div class=3D= "gmail_extra">=C2=A04 Finally, force a precise GC scan, which should guaran= tee that all no-longer referenced memory would be cleaned up at that time.<= /div> <div class=3D"gmail_extra"><br></div><div class=3D"gmail_extra">This would = actually be a very convenient working model for games. But it only works if= I know everything that was released will definitely be cleaned, otherwise = I may not be ale to allocate the next level (games often allocate all memor= y a machine has within 100k or so).</div> --20cf3074b4baf2241c04bece197c--
Apr 29 2012
prev sibling next sibling parent Sean Kelly <sean invisibleduck.org> writes:
--Apple-Mail-539B231D-B3AE-4BB2-9D01-F1F80A5C5917
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

On Apr 29, 2012, at 2:38 AM, Manu <turkeyman gmail.com> wrote:

 On 28 April 2012 18:16, Peter Alexander <peter.alexander.au gmail.com> wro=

 On Saturday, 28 April 2012 at 09:14:51 UTC, SomeDude wrote:
 On Saturday, 28 April 2012 at 09:12:23 UTC, SomeDude wrote:
=20
 Real time guarantees on a GC is not something we are going to offer anytim=

ary, with some more bells and whistles that could be borrowed from Phobos, t= his is a goal that is achievable in a foreseeable future. And both game deve= lopers and embedded programmers would be interested.
=20
 Note that Kenta Cho, who wrote fast games in D1, used this approach, and i=

=20
 I also write games in D.
=20
 My approach is this: use the GC all you want during loading or other non-i=

uring gameplay.
=20
 GC vs. manual memory allocation is a non-issue for real-time guarantees. T=

't use malloc/free during runtime because it has the same non-real-time prob= lems as using the GC. A single malloc can stall for tens of milliseconds or m= ore, and that's simply too much.
=20
 Just learn how to write code that doesn't allocate memory.
=20
 A bigger problem with GC for games is memory management i.e. controlling h=

y. Having deterministic memory usage is preferable for those cases because I= know that as soon as I delete something that the memory is available for so= mething else. I don't get that guarantee with a GC.
=20
 I think that basically sums it up.
=20
 I'm interested to know is whether using a new precise GC will guarantee AL=

 I can imagine a model in games where I could:
  1 Use the GC to allocate as much as I like during initialisation
  2 During runtime you never allocate anyway, so disable the GC (this is wh=

  3 During some clean-up, first run the logic to de-reference all things th=

  4 Finally, force a precise GC scan, which should guarantee that all no-lo=

=20
 This would actually be a very convenient working model for games. But it o=

otherwise I may not be ale to allocate the next level (games often allocate= all memory a machine has within 100k or so). For a use pattern like this, one thing that may work is to add a GC proxy im= mediately before loading a level. To unload the level, terminate that GC.=20= --Apple-Mail-539B231D-B3AE-4BB2-9D01-F1F80A5C5917 Content-Transfer-Encoding: 7bit Content-Type: text/html; charset=utf-8 <html><head></head><body bgcolor="#FFFFFF"><div>On Apr 29, 2012, at 2:38 AM, Manu &lt;<a href="mailto:turkeyman gmail.com">turkeyman gmail.com</a>&gt; wrote:<br><br></div><div></div><blockquote type="cite"><div><div class="gmail_extra"><div class="gmail_quote">On 28 April 2012 18:16, Peter Alexander <span dir="ltr">&lt;<a href="mailto:peter.alexander.au gmail.com" target="_blank">peter.alexander.au gmail.com</a>&gt;</span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On Saturday, 28 April 2012 at 09:14:51 UTC, SomeDude wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> On Saturday, 28 April 2012 at 09:12:23 UTC, SomeDude wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <br> Real time guarantees on a GC is not something we are going to offer anytime soon anyway. While a minimal library, loosely based on the C standard library, with some more bells and whistles that could be borrowed from Phobos, this is a goal that is achievable in a foreseeable future. And both game developers and embedded programmers would be interested.<br> </blockquote> <br> Note that Kenta Cho, who wrote fast games in D1, used this approach, and it worked very well for him.<br> </blockquote> <br></div></div> I also write games in D.<br> <br> My approach is this: use the GC all you want during loading or other non-interactive parts of the game and then just make sure that you don't use it during gameplay.<br> <br> GC vs. manual memory allocation is a non-issue for real-time guarantees. The simple fact of the matter is that you should be using neither. I also don't use malloc/free during runtime because it has the same non-real-time problems as using the GC. A single malloc can stall for tens of milliseconds or more, and that's simply too much.<br> <br> Just learn how to write code that doesn't allocate memory.<br> <br> A bigger problem with GC for games is memory management i.e. controlling how much memory is currently allocated, and what systems are using what memory. Having deterministic memory usage is preferable for those cases because I know that as soon as I delete something that the memory is available for something else. I don't get that guarantee with a GC.<br> </blockquote></div><br></div><div class="gmail_extra">I think that basically sums it up.</div><div class="gmail_extra"><br></div><div class="gmail_extra">I'm interested to know is whether using a new precise GC will guarantee ALL unreferenced stuff will be cleaned on any given sweep.</div> <div class="gmail_extra">I can imagine a model in games where I could:</div><div class="gmail_extra">&nbsp;1 Use the GC to allocate as much as I like during initialisation</div><div class="gmail_extra">&nbsp;2 During runtime you never allocate anyway, so disable the GC (this is when it is important to know about hidden allocations)</div> <div class="gmail_extra">&nbsp;3 During some clean-up, first run the logic to de-reference all things that are no longer required</div><div class="gmail_extra">&nbsp;4 Finally, force a precise GC scan, which should guarantee that all no-longer referenced memory would be cleaned up at that time.</div> <div class="gmail_extra"><br></div><div class="gmail_extra">This would actually be a very convenient working model for games. But it only works if I know everything that was released will definitely be cleaned, otherwise I may not be ale to allocate the next level (games often allocate all memory a machine has within 100k or so).</div> </div></blockquote><br><div>For a use pattern like this, one thing that may work is to add a GC proxy immediately before loading a level. To unload the level, terminate that GC.&nbsp;</div></body></html> --Apple-Mail-539B231D-B3AE-4BB2-9D01-F1F80A5C5917--
Apr 29 2012
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--20cf3074b41aa4c7d704bed8a478
Content-Type: text/plain; charset=UTF-8

On 29 April 2012 16:53, Sean Kelly <sean invisibleduck.org> wrote:

 On Apr 29, 2012, at 2:38 AM, Manu <turkeyman gmail.com> wrote:

 On 28 April 2012 18:16, Peter Alexander <peter.alexander.au gmail.com>wrote:

 On Saturday, 28 April 2012 at 09:14:51 UTC, SomeDude wrote:

 On Saturday, 28 April 2012 at 09:12:23 UTC, SomeDude wrote:

 Real time guarantees on a GC is not something we are going to offer
 anytime soon anyway. While a minimal library, loosely based on the C
 standard library, with some more bells and whistles that could be borrowed
 from Phobos, this is a goal that is achievable in a foreseeable future. And
 both game developers and embedded programmers would be interested.

Note that Kenta Cho, who wrote fast games in D1, used this approach, and it worked very well for him.

I also write games in D. My approach is this: use the GC all you want during loading or other non-interactive parts of the game and then just make sure that you don't use it during gameplay. GC vs. manual memory allocation is a non-issue for real-time guarantees. The simple fact of the matter is that you should be using neither. I also don't use malloc/free during runtime because it has the same non-real-time problems as using the GC. A single malloc can stall for tens of milliseconds or more, and that's simply too much. Just learn how to write code that doesn't allocate memory. A bigger problem with GC for games is memory management i.e. controlling how much memory is currently allocated, and what systems are using what memory. Having deterministic memory usage is preferable for those cases because I know that as soon as I delete something that the memory is available for something else. I don't get that guarantee with a GC.

I think that basically sums it up. I'm interested to know is whether using a new precise GC will guarantee ALL unreferenced stuff will be cleaned on any given sweep. I can imagine a model in games where I could: 1 Use the GC to allocate as much as I like during initialisation 2 During runtime you never allocate anyway, so disable the GC (this is when it is important to know about hidden allocations) 3 During some clean-up, first run the logic to de-reference all things that are no longer required 4 Finally, force a precise GC scan, which should guarantee that all no-longer referenced memory would be cleaned up at that time. This would actually be a very convenient working model for games. But it only works if I know everything that was released will definitely be cleaned, otherwise I may not be ale to allocate the next level (games often allocate all memory a machine has within 100k or so). For a use pattern like this, one thing that may work is to add a GC proxy immediately before loading a level. To unload the level, terminate that GC.

Interesting work around, although there are many other things that don't get freed from state to state, and things that are shared by both state A and B are best to keep around, save the unload/reload time of that resource (there is always lots of sharing, it adds up). Is it technically possible to have a precise GC clean up all unreferenced memory in one big pass? --20cf3074b41aa4c7d704bed8a478 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div class=3D"gmail_extra"><div class=3D"gmail_quote">On 29 April 2012 16:5= 3, Sean Kelly <span dir=3D"ltr">&lt;<a href=3D"mailto:sean invisibleduck.or= g" target=3D"_blank">sean invisibleduck.org</a>&gt;</span> wrote:<br><block= quote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc= solid;padding-left:1ex"> <div bgcolor=3D"#FFFFFF"><div><div class=3D"h5"><div>On Apr 29, 2012, at 2:= 38 AM, Manu &lt;<a href=3D"mailto:turkeyman gmail.com" target=3D"_blank">tu= rkeyman gmail.com</a>&gt; wrote:<br><br></div><div></div><blockquote type= =3D"cite"> <div><div class=3D"gmail_extra"><div class=3D"gmail_quote">On 28 April 2012= 18:16, Peter Alexander <span dir=3D"ltr">&lt;<a href=3D"mailto:peter.alexa= nder.au gmail.com" target=3D"_blank">peter.alexander.au gmail.com</a>&gt;</= span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"><div><div>On Saturday, 28 April 2012 at 09:1= 4:51 UTC, SomeDude wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> On Saturday, 28 April 2012 at 09:12:23 UTC, SomeDude wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <br> Real time guarantees on a GC is not something we are going to offer anytime= soon anyway. While a minimal library, loosely based on the C standard libr= ary, with some more bells and whistles that could be borrowed from Phobos, = this is a goal that is achievable in a foreseeable future. And both game de= velopers and embedded programmers would be interested.<br> </blockquote> <br> Note that Kenta Cho, who wrote fast games in D1, used this approach, and it= worked very well for him.<br> </blockquote> <br></div></div> I also write games in D.<br> <br> My approach is this: use the GC all you want during loading or other non-in= teractive parts of the game and then just make sure that you don&#39;t use = it during gameplay.<br> <br> GC vs. manual memory allocation is a non-issue for real-time guarantees. Th= e simple fact of the matter is that you should be using neither. I also don= &#39;t use malloc/free during runtime because it has the same non-real-time= problems as using the GC. A single malloc can stall for tens of millisecon= ds or more, and that&#39;s simply too much.<br> <br> Just learn how to write code that doesn&#39;t allocate memory.<br> <br> A bigger problem with GC for games is memory management i.e. controlling ho= w much memory is currently allocated, and what systems are using what memor= y. Having deterministic memory usage is preferable for those cases because = I know that as soon as I delete something that the memory is available for = something else. I don&#39;t get that guarantee with a GC.<br> </blockquote></div><br></div><div class=3D"gmail_extra">I think that basica= lly sums it up.</div><div class=3D"gmail_extra"><br></div><div class=3D"gma= il_extra">I&#39;m interested to know is whether using a new precise GC will= guarantee ALL unreferenced stuff will be cleaned on any given sweep.</div> <div class=3D"gmail_extra">I can imagine a model in games where I could:</d= iv><div class=3D"gmail_extra">=C2=A01 Use the GC to allocate as much as I l= ike during initialisation</div><div class=3D"gmail_extra">=C2=A02 During ru= ntime you never allocate anyway, so disable the GC (this is when it is impo= rtant to know about hidden allocations)</div> <div class=3D"gmail_extra">=C2=A03 During some clean-up, first run the logi= c to de-reference all things that are no longer required</div><div class=3D= "gmail_extra">=C2=A04 Finally, force a precise GC scan, which should guaran= tee that all no-longer referenced memory would be cleaned up at that time.<= /div> <div class=3D"gmail_extra"><br></div><div class=3D"gmail_extra">This would = actually be a very convenient working model for games. But it only works if= I know everything that was released will definitely be cleaned, otherwise = I may not be ale to allocate the next level (games often allocate all memor= y a machine has within 100k or so).</div> </div></blockquote><br></div></div><div>For a use pattern like this, one th= ing that may work is to add a GC proxy immediately before loading a level. = To unload the level, terminate that GC.=C2=A0</div></div></blockquote></div=

e are many other things that don&#39;t get freed from state to state, and t= hings that are shared by both state A and B are best to keep around, save t= he unload/reload time of that resource (there is always lots of sharing, it= adds up).</div> <div class=3D"gmail_extra">Is it technically possible to have a precise GC = clean up all unreferenced memory in one big pass?</div> --20cf3074b41aa4c7d704bed8a478--
Apr 29 2012
prev sibling next sibling parent "Tove" <tove fransson.se> writes:
On Sunday, 29 April 2012 at 22:13:22 UTC, Manu wrote:
 Is it technically possible to have a precise GC clean up all 
 unreferenced
 memory in one big pass?

yes, but unless it's also moving/compacting... one would suffer memory fragmentation... so I would imagine TempAlloc is a better fit?
Apr 29 2012
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--20cf3074b4baba15ac04bed959df
Content-Type: text/plain; charset=UTF-8

On 30 April 2012 01:24, Tove <tove fransson.se> wrote:

 On Sunday, 29 April 2012 at 22:13:22 UTC, Manu wrote:

 Is it technically possible to have a precise GC clean up all unreferenced
 memory in one big pass?

yes, but unless it's also moving/compacting... one would suffer memory fragmentation... so I would imagine TempAlloc is a better fit?

In some cases I'm comfortable with that type of fragmentation (large regularly sized resources), although that leads me to a gaping hole in D's allocation system... <OT, but still very important> There is no way to request aligned memory. I can't even specify an alignment on a user type and expect it to be aligned if I create one on the stack, let alone the heap >_< It seems I can request alignment for items within a struct, but I can't align the struct its self. In addition, a struct doesn't inherit the alignment of its aligned members, so the struct is allocated unaligned, and the aligned member fails its promise anyway. I frequently align to: 16 bytes for basically everything. This facilitates hardware simd, fast memcpy, efficient write-combining, better cache usage. 128(ish) bytes for L1 cache alignment (depending on architecture). Frequently used to guarantee ~128byte sized structs will never straddle cache lines (wasting a memory fetch/L1 eviction), and supporting predictable prefetch algorithms. 4k(ish) for texture/gpu page alignment (again, depending on architecture). Many GPU resources MUST be aligned for the GPU to access them. Swizzling is applied to aligned pages, resource allocation must match this. 4-64k virtual memory pages. Many uses. And occasionally other alignments pop up, often where they may be useful to help reduce/avoid fragmentation for instance. Sometimes I need to squat some data in a couple of low its in a pointer... requires the pointers be aligned. Obviously I can manually align my memory with various techniques, and I do, but it's rather fiddly and can also be very wasteful. One fast technique for general allocations is over-allocating by alignment-1, pasting a little header and padding the allocation. Allocating a GPU page for instance would waste another whole page just to guarantee alignment. In that case, you need to allocate a bog pool of pages and implement some pool system to dish them out, but then you need to know the precise number of pages to be allocated in advance in order not to waste memory that way. </OT> --20cf3074b4baba15ac04bed959df Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div class=3D"gmail_extra"><div class=3D"gmail_quote">On 30 April 2012 01:2= 4, Tove <span dir=3D"ltr">&lt;<a href=3D"mailto:tove fransson.se" target=3D= "_blank">tove fransson.se</a>&gt;</span> wrote:<br><blockquote class=3D"gma= il_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-lef= t:1ex"> <div class=3D"im">On Sunday, 29 April 2012 at 22:13:22 UTC, Manu wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> Is it technically possible to have a precise GC clean up all unreferenced<b= r> memory in one big pass?<br> </blockquote> <br></div> yes, but unless it&#39;s also moving/compacting... one would suffer memory = fragmentation... so I would imagine TempAlloc is a better fit?<br></blockqu= ote><div><br></div><div>In some cases I&#39;m comfortable with that type of= fragmentation (large regularly sized resources), although that leads me to= a gaping hole in D&#39;s allocation system...</div> </div></div><div class=3D"gmail_extra"><br></div><div class=3D"gmail_extra"=
&lt;OT, but still very important&gt;</div><div class=3D"gmail_extra">There=

on a user type and expect it to be aligned if I create one on the stack, l= et alone the heap &gt;_&lt;</div> <div class=3D"gmail_extra">It seems I can request alignment for items withi= n a struct, but I can&#39;t align the struct its self. In addition, a struc= t doesn&#39;t inherit the alignment of its aligned members, so the struct i= s allocated unaligned, and the aligned member fails its promise anyway.</di= v> <div class=3D"gmail_extra"><br></div><div class=3D"gmail_extra">I frequentl= y align to:</div><div class=3D"gmail_extra">16 bytes for basically everythi= ng. This facilitates hardware simd, fast memcpy, efficient write-combining,= better cache usage.</div> <div class=3D"gmail_extra">128(ish) bytes for L1 cache alignment (depending= on architecture). Frequently used to guarantee ~128byte sized structs will= never straddle cache lines (wasting a memory fetch/L1 eviction), and suppo= rting predictable prefetch algorithms.</div> <div class=3D"gmail_extra">4k(ish) for texture/gpu page alignment (again, d= epending on architecture). Many GPU resources MUST be aligned for the GPU t= o access them. Swizzling is applied to aligned pages, resource allocation m= ust match this.</div> <div class=3D"gmail_extra">4-64k virtual memory pages. Many uses.</div><div= class=3D"gmail_extra"><br></div><div class=3D"gmail_extra">And occasionall= y other alignments pop up, often where they may be useful to help reduce/av= oid fragmentation for instance.</div> <div class=3D"gmail_extra">Sometimes I need to squat some data in a couple = of low its in a pointer... requires the pointers be aligned.</div><div clas= s=3D"gmail_extra"><br></div><div class=3D"gmail_extra">Obviously I can manu= ally align my memory with various techniques, and I do,=C2=A0but it&#39;s r= ather=C2=A0fiddly=C2=A0and can also be very wasteful.</div> <div class=3D"gmail_extra">One fast technique for general allocations is=C2= =A0over-allocating by alignment-1, pasting a little header and padding the = allocation.=C2=A0Allocating a GPU page for instance would waste another who= le page just to guarantee alignment. In that case, you need to allocate a b= og pool of pages and implement some pool system to dish them out, but then = you need to know the precise number of pages to be allocated in advance in = order not to waste memory that way.</div> <div class=3D"gmail_extra">&lt;/OT&gt;</div> --20cf3074b4baba15ac04bed959df--
Apr 29 2012
prev sibling next sibling parent "Tove" <tove fransson.se> writes:
On Sunday, 29 April 2012 at 23:04:00 UTC, Manu wrote:
 In some cases I'm comfortable with that type of fragmentation 
 (large
 regularly sized resources), although that leads me to a gaping 
 hole in D's
 allocation system...

Hmmm I see, also I was thinking... since we have TLS, couldn't we abuse killing threads for fast deallocations? While adding persistent data to __gshared?
 <OT, but still very important>
 There is no way to request aligned memory. I can't even specify

I feel your pain, couldn't agree more.
Apr 29 2012
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--20cf300fac97f2ebc304bf247fa1
Content-Type: text/plain; charset=UTF-8

On 3 May 2012 12:27, Don Clugston <dac nospam.com> wrote:

 On 30/04/12 01:03, Manu wrote:

 On 30 April 2012 01:24, Tove <tove fransson.se
 <mailto:tove fransson.se>> wrote:

    On Sunday, 29 April 2012 at 22:13:22 UTC, Manu wrote:

        Is it technically possible to have a precise GC clean up all
        unreferenced
        memory in one big pass?


    yes, but unless it's also moving/compacting... one would suffer
    memory fragmentation... so I would imagine TempAlloc is a better fit?


 In some cases I'm comfortable with that type of fragmentation (large
 regularly sized resources), although that leads me to a gaping hole in
 D's allocation system...

 <OT, but still very important>
 There is no way to request aligned memory. I can't even specify an

 alignment on a user type and expect it to be aligned if I create one on
 the stack, let alone the heap >_<
 It seems I can request alignment for items within a struct, but I can't
 align the struct its self. In addition, a struct doesn't inherit the
 alignment of its aligned members, so the struct is allocated unaligned,
 and the aligned member fails its promise anyway.

Bug 2278.

Why do you suggest alignment to only 8 bytes (not 16)? MOVAPS and friends operate on 16 byte aligned data, and all non-x86 architectures are strictly 16byte aligned with no unaligned alternative possible. I'd like to see that proposal extended to arbitrary power-of-2, and to allow align(n) applied to structs/classes. --20cf300fac97f2ebc304bf247fa1 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div class=3D"gmail_extra"><div class=3D"gmail_quote">On 3 May 2012 12:27, = Don Clugston <span dir=3D"ltr">&lt;<a href=3D"mailto:dac nospam.com" target= =3D"_blank">dac nospam.com</a>&gt;</span> wrote:<br><blockquote class=3D"gm= ail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-le= ft:1ex"> <div class=3D"im">On 30/04/12 01:03, Manu wrote:<br> </div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-l= eft:1px #ccc solid;padding-left:1ex"><div class=3D"im"> On 30 April 2012 01:24, Tove &lt;<a href=3D"mailto:tove fransson.se" target= =3D"_blank">tove fransson.se</a><br></div><div><div class=3D"h5"> &lt;mailto:<a href=3D"mailto:tove fransson.se" target=3D"_blank">tove frans= son.se</a>&gt;&gt; wrote:<br> <br> =C2=A0 =C2=A0On Sunday, 29 April 2012 at 22:13:22 UTC, Manu wrote:<br> <br> =C2=A0 =C2=A0 =C2=A0 =C2=A0Is it technically possible to have a precise GC= clean up all<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0unreferenced<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0memory in one big pass?<br> <br> <br> =C2=A0 =C2=A0yes, but unless it&#39;s also moving/compacting... one would = suffer<br> =C2=A0 =C2=A0memory fragmentation... so I would imagine TempAlloc is a bet= ter fit?<br> <br> <br></div></div><div class=3D"im"> In some cases I&#39;m comfortable with that type of fragmentation (large<br=

D&#39;s allocation system...<br> <br></div><div class=3D"im"> &lt;OT, but still very important&gt;<br></div> There is no way to request aligned memory. I can&#39;t even specify an<div = class=3D"im"><br> alignment on a user type and expect it to be aligned if I create one on<br> the stack, let alone the heap &gt;_&lt;<br> It seems I can request alignment for items within a struct, but I can&#39;t= <br> align the struct its self. In addition, a struct doesn&#39;t inherit the<br=

and the aligned member fails its promise anyway.<br> </div></blockquote> <br> Bug 2278.<br> </blockquote></div><br></div><div class=3D"gmail_extra">Why do you suggest = alignment to only=C2=A08 bytes (not 16)? MOVAPS and friends operate on 16 b= yte aligned data, and all non-x86 architectures are strictly 16byte aligned= with no unaligned alternative possible.</div> <div class=3D"gmail_extra">I&#39;d like to see that proposal extended to ar= bitrary power-of-2, and to allow align(n) applied to structs/classes.</div> --20cf300fac97f2ebc304bf247fa1--
May 03 2012
prev sibling parent "Eljay" <eljay451 gmail.com> writes:
Been away for a bit.  My recap of the recent discussion, in 
reverse chronological order...

GC.  The discussion on the GC is great, *but* it may be best to 
move it to its own thread.  I don't think that the nuances of the 
GC is a critical issue preventing D becoming adopted at a company 
-- even if those nuances (e.g., memory fragmentation, 
BlkAttr.NO_SCAN, false pointers, 10 second garbage collection, 
room for improvement) are very important to fully grok.

J2EE.  Using D instead of J2EE is an interesting notion.  
Companies that are invested in using J2EE probably are not 
amenable to changing from JVM, so D would be viable for 
J2EE-centric enterprise environments if it could be compiled to 
Java Bytecode.  (The J2EE infrastructure provides a lot of 
administration, monitoring, and management facilities.)

Games.  As Peter indicated, and I assume Kenta Cho would 
whole-heartedly agree, for high-performance games the GC (or even 
malloc/free) can be avoided.  D is a fully viable language for 
high-performance games.  Awesome!

License.  The discussion on the licenses, as far as I can tell, 
impacts several things, and merits having further discussion in 
its own thread.  Its great to see that D2 compilers are coming 
either as stock part of some Linux distros, or as an easily 
obtainable package[1].
[1] alas far less great, since the "slight" increase to barrier 
to entry is actually quite high.  Back before Ruby and Python 
were stock components, I've seen people who prefer Python or 
prefer Ruby use Perl instead just because they could rely on it 
being there -- even though pulling down Python or Ruby was a snap.

Arrays.  Thank you for bringing to my attention the GC 
implications of using the built-in arrays and slicing.  I think 
that having D-based template C++ STL/BOOST-like alternatives that 
have different (non-GC) memory requirements makes sense now.  I 
don't think this is a show-stopper for D becoming adopted at a 
company.
May 07 2012