www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - [Issue 8993] New: Implement unique references/isolated memory

reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993

           Summary: Implement unique references/isolated memory
           Product: D
           Version: D2
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: enhancement
          Priority: P2
         Component: DMD
        AssignedTo: nobody puremagic.com
        ReportedBy: sludwig outerproduct.org


--- Comment #0 from Sönke Ludwig <sludwig outerproduct.org> 2012-11-10 06:40:09
PST ---
'unique' or 'isolated' is a recursive type qualifier which enforces that there
is only a single reference to a given piece of memory, and that the memory
itself is only pointing to other data that is either immutable, or only
referenced within the same memory cluster.

Any value/reference that is tagged this way can safely be converted to
immutable or can safely be passed (moved) to another thread. In conjunction
with (weakly) pure functions it is possible to operate almost freely on a
unique value, because it can be guaranteed that no visible additional
references can be created.

The main advantages of this approach are:

 - Provides a powerful alternative to safely pass mutable data between threads
without requiring locks or synchronization primitives (synchronized/shared)

 - Construction of immutable objects can be done in a natural procedural
fashion and not only within the object constructor or within a pure function

 - Fully compatible with the current type system, builds nicely on top of the
current base

 - The system is proven both in theory and in practice

Microsoft has released a paper describing the system in form of an extension to
C#. The paper includes formal proofs for the soundness of the system and
mentions a multi-million lines of code project for which it is used:

http://research.microsoft.com/pubs/170528/msr-tr-2012-79.pdf

D specific and in addition to the Microsoft paper, a solution to support
partitioning of a uniqe array with unique slices as the result is desirable.
But this can be done as a simple library function if needed.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 10 2012
next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993


Alex Rĝnne Petersen <alex lycus.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |alex lycus.org


--- Comment #1 from Alex Rĝnne Petersen <alex lycus.org> 2012-11-10 15:46:00
CET ---
I think it's a bit too late to add new keywords to the language, other than
ones prefixed with "__", and we certainly wouldn't want something this useful
to be subject to such annoying-to-type keywords...

This seems like a thing for D3 *if* D3 ever becomes a thing.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 10 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993


bearophile_hugs eml.cc changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bearophile_hugs eml.cc


--- Comment #2 from bearophile_hugs eml.cc 2012-11-10 08:24:06 PST ---
Writing a detailed DEP requires lot of work, but maybe for a request this
complex it's unavoidable.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 10 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993



--- Comment #3 from Sönke Ludwig <sludwig outerproduct.org> 2012-11-10 08:38:03
PST ---
(In reply to comment #1)
 I think it's a bit too late to add new keywords to the language, other than
 ones prefixed with "__", and we certainly wouldn't want something this useful
 to be subject to such annoying-to-type keywords...
 
 This seems like a thing for D3 *if* D3 ever becomes a thing.

Even with __ it would be better than nothing. I think it's almost safe to say that current D is almost unusable in conjunction with immutable/shared (objects) and that's ironically one of the areas where it tries to shine. I have tried numerous times to cleanly reimplement certain multithreaded functionality using it and _never_ got it running without just so many casts that the result was not only as unsafe/unchecked as the original, but also unreadable und badly usable. I apologize in advance for the following (general) rant: The attitude of implementing ad-hoc solutions, not listening to valid objections and then declaring the language as stable is a very foolish development approach. The way shared was introduced was not the only thing in the past that went this way. Things like these made me really frustrated with D multiple times, to the point where I was seriously looking out for alternatives - the end result was pure disregard/disinterest for the corresponding language parts. And I think that there is quite a number of people who just gave up on either the language or on certain parts, and the resulting silence creates a dangerous deceived feeling of statisfaction. The current mode of development is shizophrenic, stabilizing an obviously unfinished language and, from time to time, introducing new, breaking things anyway. There absolutely needs to be a way to still let innovation and refinement happen, be it in a separate branch (called 3.x or something else doesn't matter) or in the form of allowing new features as long as they are backwards-compatible (e.g. using the __ way). Of course the constant breakage of the past is bad, but stagnation at this point could possibly be the start of a slow, starving death. Sorry again, but I've seen D make so many mistakes in the past and that it has done a lot more things right is no excuse for that. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Nov 10 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993



--- Comment #4 from Alex Rĝnne Petersen <alex lycus.org> 2012-11-10 17:47:59
CET ---
Yeah, no doubt shared is a disaster in general (I have posted several rants
about this in particular on the NG). We're effectively doing a lot of false
advertising with that type qualifier in particular, and people are writing code
as if it results in atomic operations (and then their code will break on !x86).

And immutable/const are only realistically usable with non-class types. I feel
you're spot on with the point about perceived satisfaction.

But it's not me you have to convince. It's Walter and Andrei...

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 10 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993



--- Comment #5 from bearophile_hugs eml.cc 2012-11-10 09:48:35 PST ---
(In reply to comment #3)

 I apologize in advance for the following (general) rant:

I agree with what you say. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Nov 10 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993



--- Comment #6 from bearophile_hugs eml.cc 2012-11-10 09:50:22 PST ---
(In reply to comment #5)

 I agree with what you say.

But in the end this enhancement request is not enough. A detailed DEP is probably necessary. You don't have to write it down all yourself, of course. Once the DEP is written, you will be in a better position to rant :-) -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Nov 10 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993



--- Comment #7 from Peter Alexander <peter.alexander.au gmail.com> 2012-11-10
10:04:02 PST ---
Just want to chime in to say that I 100% agree with Sönke and Alex.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 10 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993



--- Comment #8 from Sönke Ludwig <sludwig outerproduct.org> 2012-11-10 10:38:42
PST ---
(In reply to comment #6)
 (In reply to comment #5)
 
 I agree with what you say.

But in the end this enhancement request is not enough. A detailed DEP is probably necessary. You don't have to write it down all yourself, of course. Once the DEP is written, you will be in a better position to rant :-)

Agreed, a DIP is probably in order for this change. It would probably be enough to concentrate on the simple parts first (implicit isolated->immutable conversion and free psasing of isolated values between threads) and then implement further relaxations bit by bit (e.g. implicit scoped isolated->writable->isolated conversion). Allocating time for a proper DIP is difficult for me currently, but I guess it could be marked as work-in-progress in the wiki so that it can be completed step by step? -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Nov 10 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993



--- Comment #9 from bearophile_hugs eml.cc 2012-11-10 11:11:35 PST ---
(In reply to comment #8)

 but I guess it could be marked as work-in-progress
 in the wiki so that it can be completed step by step?

I think this is OK. In the paper they say the Microsoft programmers have implemented that system one piece at a time. So take a look at those implementation steps (avoiding their eventual mistakes). -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Nov 10 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993



--- Comment #10 from Sönke Ludwig <sludwig outerproduct.org> 2012-11-10
11:22:20 PST ---
(In reply to comment #9)
 (In reply to comment #8)
 
 but I guess it could be marked as work-in-progress
 in the wiki so that it can be completed step by step?

I think this is OK. In the paper they say the Microsoft programmers have implemented that system one piece at a time. So take a look at those implementation steps (avoiding their eventual mistakes).

Oh, I meant more the process of writing the actual DIP in that sentence (I never did anything on the wiki and don't know the usual process for editing a DIP). So my plan was basically to only include the first step of the implementation into the DIP and then file the rest as enhancement requests should it get implemented at some point. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Nov 10 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993


Jacob Carlborg <doob me.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |doob me.com


--- Comment #11 from Jacob Carlborg <doob me.com> 2012-11-11 01:54:31 PST ---
I recommend you take a look at some blog posts from Bartosz Milewski. He was
involved for a while in the D community and had some ideas about race-free
multithreading with the help of a type system supporting using "unique" and
"lent". Search in the newsgroups archives for "Bartosz Milewski".

http://bartoszmilewski.com/2009/05/26/race-free-multithreading/

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 11 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993



--- Comment #12 from Sönke Ludwig <sludwig outerproduct.org> 2012-11-11
04:49:41 PST ---
(In reply to comment #11)
 I recommend you take a look at some blog posts from Bartosz Milewski. He was
 involved for a while in the D community and had some ideas about race-free
 multithreading with the help of a type system supporting using "unique" and
 "lent". Search in the newsgroups archives for "Bartosz Milewski".
 
 http://bartoszmilewski.com/2009/05/26/race-free-multithreading/

I've read those back then. Bartosz' system was very powerful but unfortunately complex to implement and had some implications on performance. The good thing with the MS approach is that it accomplishes an important part of it without requiring explicit owner tracking. It surely doesn't fix 'shared' the way that Bartosz' system would, but at least it alleviates the need for it in quite some places.(In reply to comment #11) -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Nov 11 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993


Rob Jacques <sandford jhu.edu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |sandford jhu.edu


--- Comment #13 from Rob Jacques <sandford jhu.edu> 2012-11-11 23:25:31 PST ---
(In reply to comment #8)
 implement further relaxations bit by bit (e.g. implicit scoped
 isolated->writable->isolated conversion). 

To clarify for those who didn't read the paper, the implicit conversion of writable->isolated only applies to the return value of a moderately pure function: (a pure function whose arguments are only isolated or immutable) So it is similar to the implicit conversion of the return of a strongly pure function from mutable to immutable. -- Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email ------- You are receiving this mail because: -------
Nov 11 2012
prev sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993



--- Comment #14 from bearophile_hugs eml.cc 2013-02-28 18:37:48 PST ---
See also:

https://github.com/D-Programming-Language/dmd/pull/1700

http://wiki.dlang.org/DIP29

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Feb 28 2013