www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - [Issue 8993] New: Implement unique references/isolated memory

reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993

           Summary: Implement unique references/isolated memory
           Product: D
           Version: D2
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: enhancement
          Priority: P2
         Component: DMD
        AssignedTo: nobody puremagic.com
        ReportedBy: sludwig outerproduct.org



PST ---
'unique' or 'isolated' is a recursive type qualifier which enforces that there
is only a single reference to a given piece of memory, and that the memory
itself is only pointing to other data that is either immutable, or only
referenced within the same memory cluster.

Any value/reference that is tagged this way can safely be converted to
immutable or can safely be passed (moved) to another thread. In conjunction
with (weakly) pure functions it is possible to operate almost freely on a
unique value, because it can be guaranteed that no visible additional
references can be created.

The main advantages of this approach are:

 - Provides a powerful alternative to safely pass mutable data between threads
without requiring locks or synchronization primitives (synchronized/shared)

 - Construction of immutable objects can be done in a natural procedural
fashion and not only within the object constructor or within a pure function

 - Fully compatible with the current type system, builds nicely on top of the
current base

 - The system is proven both in theory and in practice

Microsoft has released a paper describing the system in form of an extension to

mentions a multi-million lines of code project for which it is used:

http://research.microsoft.com/pubs/170528/msr-tr-2012-79.pdf

D specific and in addition to the Microsoft paper, a solution to support
partitioning of a uniqe array with unique slices as the result is desirable.
But this can be done as a simple library function if needed.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 10 2012
next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993


Alex Rĝnne Petersen <alex lycus.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |alex lycus.org



CET ---
I think it's a bit too late to add new keywords to the language, other than
ones prefixed with "__", and we certainly wouldn't want something this useful
to be subject to such annoying-to-type keywords...

This seems like a thing for D3 *if* D3 ever becomes a thing.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 10 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993


bearophile_hugs eml.cc changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bearophile_hugs eml.cc



Writing a detailed DEP requires lot of work, but maybe for a request this
complex it's unavoidable.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 10 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993




PST ---


 I think it's a bit too late to add new keywords to the language, other than
 ones prefixed with "__", and we certainly wouldn't want something this useful
 to be subject to such annoying-to-type keywords...
 
 This seems like a thing for D3 *if* D3 ever becomes a thing.


Even with __ it would be better than nothing. I think it's almost safe to say
that current D is almost unusable in conjunction with immutable/shared
(objects) and that's ironically one of the areas where it tries to shine. I
have tried numerous times to cleanly reimplement certain multithreaded
functionality using it and _never_ got it running without just so many casts
that the result was not only as unsafe/unchecked as the original, but also
unreadable und badly usable.


I apologize in advance for the following (general) rant:

The attitude of implementing ad-hoc solutions, not listening to valid
objections and then declaring the language as stable is a very foolish
development approach. The way shared was introduced was not the only thing in
the past that went this way.

Things like these made me really frustrated with D multiple times, to the point
where I was seriously looking out for alternatives - the end result was pure
disregard/disinterest for the corresponding language parts. And I think that
there is quite a number of people who just gave up on either the language or on
certain parts, and the resulting silence creates a dangerous deceived feeling
of statisfaction.

The current mode of development is shizophrenic, stabilizing an obviously
unfinished language and, from time to time, introducing new, breaking things
anyway. There absolutely needs to be a way to still let innovation and
refinement happen, be it in a separate branch (called 3.x or something else
doesn't matter) or in the form of allowing new features as long as they are
backwards-compatible (e.g. using the __ way). Of course the constant breakage
of the past is bad, but stagnation at this point could possibly be the start of
a slow, starving death.

Sorry again, but I've seen D make so many mistakes in the past and that it has
done a lot more things right is no excuse for that.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 10 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993




CET ---
Yeah, no doubt shared is a disaster in general (I have posted several rants
about this in particular on the NG). We're effectively doing a lot of false
advertising with that type qualifier in particular, and people are writing code
as if it results in atomic operations (and then their code will break on !x86).

And immutable/const are only realistically usable with non-class types. I feel
you're spot on with the point about perceived satisfaction.

But it's not me you have to convince. It's Walter and Andrei...

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 10 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993







 I apologize in advance for the following (general) rant:


I agree with what you say.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 10 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993







 I agree with what you say.


But in the end this enhancement request is not enough. A detailed DEP is
probably necessary. You don't have to write it down all yourself, of course.
Once the DEP is written, you will be in a better position to rant :-)

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 10 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993




10:04:02 PST ---
Just want to chime in to say that I 100% agree with Sönke and Alex.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 10 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993




PST ---



 

 I agree with what you say.


 
 But in the end this enhancement request is not enough. A detailed DEP is
 probably necessary. You don't have to write it down all yourself, of course.
 Once the DEP is written, you will be in a better position to rant :-)


Agreed, a DIP is probably in order for this change. It would probably be enough
to concentrate on the simple parts first (implicit isolated->immutable
conversion and free psasing of isolated values between threads) and then
implement further relaxations bit by bit (e.g. implicit scoped
isolated->writable->isolated conversion). Allocating time for a proper DIP is
difficult for me currently, but I guess it could be marked as work-in-progress
in the wiki so that it can be completed step by step?

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 10 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993







 but I guess it could be marked as work-in-progress
 in the wiki so that it can be completed step by step?


I think this is OK.

In the paper they say the Microsoft programmers have implemented that system
one piece at a time. So take a look at those implementation steps (avoiding
their eventual mistakes).

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 10 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993




11:22:20 PST ---



 

 but I guess it could be marked as work-in-progress
 in the wiki so that it can be completed step by step?


 
 I think this is OK.
 
 In the paper they say the Microsoft programmers have implemented that system
 one piece at a time. So take a look at those implementation steps (avoiding
 their eventual mistakes).


Oh, I meant more the process of writing the actual DIP in that sentence (I
never did anything on the wiki and don't know the usual process for editing a
DIP). So my plan was basically to only include the first step of the
implementation into the DIP and then file the rest as enhancement requests
should it get implemented at some point.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 10 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993


Jacob Carlborg <doob me.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |doob me.com



I recommend you take a look at some blog posts from Bartosz Milewski. He was
involved for a while in the D community and had some ideas about race-free
multithreading with the help of a type system supporting using "unique" and
"lent". Search in the newsgroups archives for "Bartosz Milewski".

http://bartoszmilewski.com/2009/05/26/race-free-multithreading/

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 11 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993




04:49:41 PST ---


 I recommend you take a look at some blog posts from Bartosz Milewski. He was
 involved for a while in the D community and had some ideas about race-free
 multithreading with the help of a type system supporting using "unique" and
 "lent". Search in the newsgroups archives for "Bartosz Milewski".
 
 http://bartoszmilewski.com/2009/05/26/race-free-multithreading/


I've read those back then. Bartosz' system was very powerful but unfortunately
complex to implement and had some implications on performance. The good thing
with the MS approach is that it accomplishes an important part of it without
requiring explicit owner tracking. It surely doesn't fix 'shared' the way that
Bartosz' system would, but at least it alleviates the need for it in quite some


-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 11 2012
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993


Rob Jacques <sandford jhu.edu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |sandford jhu.edu





 implement further relaxations bit by bit (e.g. implicit scoped
 isolated->writable->isolated conversion). 


To clarify for those who didn't read the paper, the implicit conversion of
writable->isolated only applies to the return value of a moderately pure
function: (a pure function whose arguments are only isolated or immutable) So
it is similar to the implicit conversion of the return of a strongly pure
function from mutable to immutable.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Nov 11 2012
prev sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=8993




See also:

https://github.com/D-Programming-Language/dmd/pull/1700

http://wiki.dlang.org/DIP29

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Feb 28 2013