www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Concurrency architecture for D2

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
I think we are now in the position of defining a solid set of 
concurrency primitives for D. This follows many months of mulling over 
models and options.

It would be great to open the participation to the design as broadly as 
possible, but I think it's realistic to say we won't be able to get 
things done on the newsgroup. When we discuss a topic around here, 
there's plenty of good ideas but also the inevitable bikeshed 
discussions, explanations being asked, explanations being given, and 
other sources of noise. We simply don't have the time to deal with all 
that - the time is short and we only have one shot at this.

That's why I'm thinking of creating a mailing list or maybe another 
group for this. Any ideas on what would be the best approach? I also 
want to gauge interest from threading experts who'd like to participate. 
Please advise: (a) whether you would like to participate to the design; 
(b) keep discussions on the general group; (c) create a separate 
newsgroup; (d) create a mailing list. The latter would have open enrollment.


Andrei
Dec 27 2009
next sibling parent Jason House <jason.james.house gmail.com> writes:
I'd like to participate. Both email and newsgroup are fine. Even IRC or other
Internet chatroom would be ok too. Past concurrency/shared discussions on this
newsgroup have not suffered from bikeshed issues. I think it's all been too
theoretical or outside moat programmers' experience. There's almost no keywords
to debate anyway.

Andrei Alexandrescu Wrote:

 I think we are now in the position of defining a solid set of 
 concurrency primitives for D. This follows many months of mulling over 
 models and options.
 
 It would be great to open the participation to the design as broadly as 
 possible, but I think it's realistic to say we won't be able to get 
 things done on the newsgroup. When we discuss a topic around here, 
 there's plenty of good ideas but also the inevitable bikeshed 
 discussions, explanations being asked, explanations being given, and 
 other sources of noise. We simply don't have the time to deal with all 
 that - the time is short and we only have one shot at this.
 
 That's why I'm thinking of creating a mailing list or maybe another 
 group for this. Any ideas on what would be the best approach? I also 
 want to gauge interest from threading experts who'd like to participate. 
 Please advise: (a) whether you would like to participate to the design; 
 (b) keep discussions on the general group; (c) create a separate 
 newsgroup; (d) create a mailing list. The latter would have open enrollment.
 
 
 Andrei

Dec 27 2009
prev sibling next sibling parent reply retard <re tard.com.invalid> writes:
Sun, 27 Dec 2009 14:32:52 -0600, Andrei Alexandrescu wrote:

 I think we are now in the position of defining a solid set of
 concurrency primitives for D. This follows many months of mulling over
 models and options.
 
 It would be great to open the participation to the design as broadly as
 possible, but I think it's realistic to say we won't be able to get
 things done on the newsgroup. When we discuss a topic around here,
 there's plenty of good ideas but also the inevitable bikeshed
 discussions, explanations being asked, explanations being given, and
 other sources of noise. We simply don't have the time to deal with all
 that - the time is short and we only have one shot at this.
 
 That's why I'm thinking of creating a mailing list or maybe another
 group for this. Any ideas on what would be the best approach? I also
 want to gauge interest from threading experts who'd like to participate.
 Please advise: (a) whether you would like to participate to the design;
 (b) keep discussions on the general group; (c) create a separate
 newsgroup; (d) create a mailing list. The latter would have open
 enrollment.

Have the discussions here ever led to a conclusion and get implemented? You should at least disallow posting via the web interface - those broken threads start to annoy some people. While you're at it, I, for one, welcome a new concurrency architecture that scales nicely when I want optimal array processing on i486, PII/SSE, P4/HT, Core i7/HT/SSE 4.2, and perhaps also on Beowulf (takes care of network latencies). Would it use the new immutable actor/message model you're building?
Dec 27 2009
next sibling parent reply Jason House <jason.james.house gmail.com> writes:
retard Wrote:

 Sun, 27 Dec 2009 14:32:52 -0600, Andrei Alexandrescu wrote:
 
 I think we are now in the position of defining a solid set of
 concurrency primitives for D. This follows many months of mulling over
 models and options.
 
 It would be great to open the participation to the design as broadly as
 possible, but I think it's realistic to say we won't be able to get
 things done on the newsgroup. When we discuss a topic around here,
 there's plenty of good ideas but also the inevitable bikeshed
 discussions, explanations being asked, explanations being given, and
 other sources of noise. We simply don't have the time to deal with all
 that - the time is short and we only have one shot at this.
 
 That's why I'm thinking of creating a mailing list or maybe another
 group for this. Any ideas on what would be the best approach? I also
 want to gauge interest from threading experts who'd like to participate.
 Please advise: (a) whether you would like to participate to the design;
 (b) keep discussions on the general group; (c) create a separate
 newsgroup; (d) create a mailing list. The latter would have open
 enrollment.

Have the discussions here ever led to a conclusion and get implemented?

DIP2 is a great example of something designed on this newsgroup that got implemented very recently. It went from newsgroup to bugzilla to DIP to implementation. T[new] is another one that almost made it.
 You should at least disallow posting via the web interface - those broken 
 threads start to annoy some people.

Do my posts show up as broken threads? I use the web interface almost exclusively.
Dec 27 2009
parent reply Robert Clipsham <robert octarineparrot.com> writes:
On 27/12/09 23:56, Jason House wrote:
 You should at least disallow posting via the web interface - those broken
 threads start to annoy some people.

Do my posts show up as broken threads? I use the web interface almost exclusively.

Which web interface do you use? there are 2 of them, maybe one of them doesn't break it?
Dec 27 2009
parent Jason House <jason.james.house gmail.com> writes:
Robert Clipsham Wrote:

 On 27/12/09 23:56, Jason House wrote:
 You should at least disallow posting via the web interface - those broken
 threads start to annoy some people.

Do my posts show up as broken threads? I use the web interface almost exclusively.

Which web interface do you use? there are 2 of them, maybe one of them doesn't break it?

Here is the URL I used to view your message: http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=104129
Dec 27 2009
prev sibling parent retard <re tard.com.invalid> writes:
Sun, 27 Dec 2009 18:56:09 -0500, Jason House wrote:

 retard Wrote:
 
 Sun, 27 Dec 2009 14:32:52 -0600, Andrei Alexandrescu wrote:
 
 I think we are now in the position of defining a solid set of
 concurrency primitives for D. This follows many months of mulling
 over models and options.
 
 It would be great to open the participation to the design as broadly
 as possible, but I think it's realistic to say we won't be able to
 get things done on the newsgroup. When we discuss a topic around
 here, there's plenty of good ideas but also the inevitable bikeshed
 discussions, explanations being asked, explanations being given, and
 other sources of noise. We simply don't have the time to deal with
 all that - the time is short and we only have one shot at this.
 
 That's why I'm thinking of creating a mailing list or maybe another
 group for this. Any ideas on what would be the best approach? I also
 want to gauge interest from threading experts who'd like to
 participate. Please advise: (a) whether you would like to participate
 to the design; (b) keep discussions on the general group; (c) create
 a separate newsgroup; (d) create a mailing list. The latter would
 have open enrollment.

Have the discussions here ever led to a conclusion and get implemented?

DIP2 is a great example of something designed on this newsgroup that got implemented very recently. It went from newsgroup to bugzilla to DIP to implementation. T[new] is another one that almost made it.

Ok, good to know.
 You should at least disallow posting via the web interface - those
 broken threads start to annoy some people.

Do my posts show up as broken threads? I use the web interface almost exclusively.

This one didn't break the thread. But sometimes it does break.
Dec 27 2009
prev sibling next sibling parent Robert Clipsham <robert octarineparrot.com> writes:
On 27/12/09 20:32, Andrei Alexandrescu wrote:
 I think we are now in the position of defining a solid set of
 concurrency primitives for D. This follows many months of mulling over
 models and options.

Excellent! The last time I had a real play with D2 this is what I found was missing, particularly when a lot of the newer features are built with concurrency in mind.
 It would be great to open the participation to the design as broadly as
 possible, but I think it's realistic to say we won't be able to get
 things done on the newsgroup. When we discuss a topic around here,
 there's plenty of good ideas but also the inevitable bikeshed
 discussions, explanations being asked, explanations being given, and
 other sources of noise. We simply don't have the time to deal with all
 that - the time is short and we only have one shot at this.

You're most probably right about this. It would be nice for it to be as open as possible, you don't want it ending up with bikesheds all over the place though, particularly at this early stage. Maybe a sort of guided openness could be used? If you go with another newsgroup it won't matter about the noise, and you could go through the design process in a nice guided manner, getting people's initial ideas, voting on them, discussing pro's/cons etc, that way you get something reasonably workable before you get to the bikesheds. If you let everyone dive in straight off I don't think much will come out of it
 That's why I'm thinking of creating a mailing list or maybe another
 group for this. Any ideas on what would be the best approach? I also
 want to gauge interest from threading experts who'd like to participate.
 Please advise: (a) whether you would like to participate to the design;
 (b) keep discussions on the general group; (c) create a separate
 newsgroup; (d) create a mailing list. The latter would have open
 enrollment.

(a) I doubt I'd be much use here, I might have a comment here and there to throw in though (b..$) b or c are the preferable options for me, they're easier for me to follow and require little or no setup... excuse my laziness!
Dec 27 2009
prev sibling next sibling parent Kevin Bealer <kevinbealer gmail.com> writes:
Andrei Alexandrescu Wrote:

 I think we are now in the position of defining a solid set of 
 concurrency primitives for D. This follows many months of mulling over 
 models and options.
 
 It would be great to open the participation to the design as broadly as 
 possible, but I think it's realistic to say we won't be able to get 
 things done on the newsgroup. When we discuss a topic around here, 
 there's plenty of good ideas but also the inevitable bikeshed 
 discussions, explanations being asked, explanations being given, and 
 other sources of noise. We simply don't have the time to deal with all 
 that - the time is short and we only have one shot at this.
 
 That's why I'm thinking of creating a mailing list or maybe another 
 group for this. Any ideas on what would be the best approach? I also 
 want to gauge interest from threading experts who'd like to participate. 
 Please advise: (a) whether you would like to participate to the design; 
 (b) keep discussions on the general group; (c) create a separate 
 newsgroup; (d) create a mailing list. The latter would have open enrollment.
 
 
 Andrei

I'd love to follow the discussion. I'm not sure whether I have all that much expertise for contributing to the "core debate". It seems like you and Walter have had a pattern where you discuss things among yourself and post the occasional "here's our thinking" to a group like this for the rest of us; I'd like to at least hear about the cool stuff periodically, even if the discussion here is just chewing the scraps from the real table. That said, I'll throw out one high level thought on the topic: As for concurrency itself, my gut reaction is that I'd hope there would be mutex + thread. Obviously you want something better, and I look forward to hearing all the cool stuff about it; but as clunky as we all know mutex + thread are, it would be good to have them in some form. One of the downsides of inventing / using a "new thing" is that when all the existing systems use a particular programming paradigm it becomes hard to integrate with them if you can't at least emulate that paradigm in a more or less straightforward way. If someone is wrapping OpenGL in their D code, using a C library (just to pick an example, I don't know much OpenGL), it will be very hard for them to bend OpenGL into an immutable / message passing approach if it was designed for the land of thread + mutex. I think that one of the big lessons of what might later be called the "silver bullet phase" of CompSci, is that there should be a marketplace where the various paradigms must compete. If you build a new paradigm for this it is best if it is a top predator in an ecosystem of ideas, not a new planet with only a single monoculture. All paradigms have a place; if a paradigm is used, there is probably a niche for that approach. If it exists, it must have value. Except for bubble-sort. Kevin
Dec 28 2009
prev sibling next sibling parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2009-12-27 15:32:52 -0500, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:

 I think we are now in the position of defining a solid set of 
 concurrency primitives for D. This follows many months of mulling over 
 models and options.
 
 It would be great to open the participation to the design as broadly as 
 possible, but I think it's realistic to say we won't be able to get 
 things done on the newsgroup. When we discuss a topic around here, 
 there's plenty of good ideas but also the inevitable bikeshed 
 discussions, explanations being asked, explanations being given, and 
 other sources of noise. We simply don't have the time to deal with all 
 that - the time is short and we only have one shot at this.
 
 That's why I'm thinking of creating a mailing list or maybe another 
 group for this. Any ideas on what would be the best approach? I also 
 want to gauge interest from threading experts who'd like to 
 participate. Please advise: (a) whether you would like to participate 
 to the design; (b) keep discussions on the general group; (c) create a 
 separate newsgroup; (d) create a mailing list. The latter would have 
 open enrollment.

I think it should be as open as possible. If done in a separate smaller group, it may be a good idea to post reports to the general newsgroup more or less regularly so that those who cannot participate in the detailed discussions have an idea of where it's going, and also to get more general input. About the bikeshed issue, I'm not sure how much those bikeshed discussions are slowing down the more important ones, but they often start from legitimate real, often syntactic, issues. Those discussions shouldn't be avoided just because everyone has an opinion. But perhaps regular reports to the general newsgroup would help confining them there. I'd be in favor of creating a newsgroup for concurrency, and I'll probably want to participate a little too, although I'm not sure how much yet. -- Michel Fortin michel.fortin michelf.com http://michelf.com/
Dec 28 2009
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Michel Fortin wrote:
 On 2009-12-27 15:32:52 -0500, Andrei Alexandrescu 
 <SeeWebsiteForEmail erdani.org> said:
 
 I think we are now in the position of defining a solid set of 
 concurrency primitives for D. This follows many months of mulling over 
 models and options.

 It would be great to open the participation to the design as broadly 
 as possible, but I think it's realistic to say we won't be able to get 
 things done on the newsgroup. When we discuss a topic around here, 
 there's plenty of good ideas but also the inevitable bikeshed 
 discussions, explanations being asked, explanations being given, and 
 other sources of noise. We simply don't have the time to deal with all 
 that - the time is short and we only have one shot at this.

 That's why I'm thinking of creating a mailing list or maybe another 
 group for this. Any ideas on what would be the best approach? I also 
 want to gauge interest from threading experts who'd like to 
 participate. Please advise: (a) whether you would like to participate 
 to the design; (b) keep discussions on the general group; (c) create a 
 separate newsgroup; (d) create a mailing list. The latter would have 
 open enrollment.

I think it should be as open as possible. If done in a separate smaller group, it may be a good idea to post reports to the general newsgroup more or less regularly so that those who cannot participate in the detailed discussions have an idea of where it's going, and also to get more general input.

That's obviously the best way to go, but there are a couple of circumstances that make that more difficult. 1. Chapter drafts will be the basis for discussion, but understandably the publisher does not allow me to freely distribute them. 2. Time. There are regulars on this group that have a "when in doubt, make them sweat" policy. I think it's a very good and gainful attitude for everyone involved, and I generally enjoy discussing this or that idea because it helps me and others gain a better understanding, but this time there won't be much time for discussions of the form: a) Poster: "Subtle issue X sounds like a bad idea. I don't agree with it." b) <Long argumentation back and forth.> c) Poster: "I stay unconvinced." or "That makes sense." There will be very little time for anything like this, particularly if explaining X requires a fair amount of background building. Building a shared vision is very difficult among only a small group of people, and doing so for a larger group will be an enormous drag. I feel very lucky that Walter and I share views most of the time (except, of course, when he's wrong :o)).
 About the bikeshed issue, I'm not sure how much those bikeshed 
 discussions are slowing down the more important ones, but they often 
 start from legitimate real, often syntactic, issues. Those discussions 
 shouldn't be avoided just because everyone has an opinion. But perhaps 
 regular reports to the general newsgroup would help confining them there.
 
 I'd be in favor of creating a newsgroup for concurrency, and I'll 
 probably want to participate a little too, although I'm not sure how 
 much yet.

OK, let's see what Walter thinks. Andrei
Dec 28 2009
parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2009-12-28 11:20:53 -0500, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:

 Michel Fortin wrote:
 
 I think it should be as open as possible. If done in a separate smaller 
 group, it may be a good idea to post reports to the general newsgroup 
 more or less regularly so that those who cannot participate in the 
 detailed discussions have an idea of where it's going, and also to get 
 more general input.

That's obviously the best way to go, but there are a couple of circumstances that make that more difficult. 1. Chapter drafts will be the basis for discussion, but understandably the publisher does not allow me to freely distribute them.

I see. I find that unfortunate, because this seems to imply it'll need to be a closed group, not only closed regarding the participants but also regarding external observers (because the basis of the discussion will be hidden from them). Often you can evaluate how good something is not only by looking at the final result, but by looking at the process that led to it. Having an archive of the past is often very useful to check why some things have been made that way. But that works best if the discussion are done in the open.
 2. Time. There are regulars on this group that have a "when in doubt, 
 make them sweat" policy. I think it's a very good and gainful attitude 
 for everyone involved, and I generally enjoy discussing this or that 
 idea because it helps me and others gain a better understanding, but 
 this time there won't be much time for discussions of the form:
 
 a) Poster: "Subtle issue X sounds like a bad idea. I don't agree with it."
 
 b) <Long argumentation back and forth.>
 
 c) Poster: "I stay unconvinced." or "That makes sense."
 
 There will be very little time for anything like this, particularly if 
 explaining X requires a fair amount of background building.
 
 Building a shared vision is very difficult among only a small group of 
 people, and doing so for a larger group will be an enormous drag. I 
 feel very lucky that Walter and I share views most of the time (except, 
 of course, when he's wrong :o)).

I think it'd be a good idea to set an objective for the group for that limited time frame. This way if the discussion deviates towards something off-topic it's easy to suggest posting to the general or the D.learn newsgroup instead, or if the discussion is draining taking too much time in explanations it's easy to justify postponing that to later in order to fulfill the more pressing objectives. Just be careful not setting the goal too broadly. Also keep in mind that we don't really need a shared vision among everyone. What's needed is someone who takes the decisions. Discussion is only needed to help that person take the right decisions. Although consensus among all members certainly boosts the decider self-confidence, it is not required, and not necessarily desirable either. A consensus among only a few key people is all that is needed, and this has little to do with who is allowed to raise issues and propose solutions. And those few key people are the implementors, because if they don't agree the result isn't gonna be pretty. (This reminds me pretty much of how HTML5 is being developed right now: Ian Hickson, the spec editor, decides what goes in and out. But he always seeks the broadest consensus among the major browser vendors because in the end it's them who decide what they want to implement, and if they refuse to implement the spec, the spec is useless. This doesn't prevent the WHATWG mailing list from being open to all. I did some research a few years ago, posted the results on the list and after some discussion it was used to make the <figure> element what it is now. I wouldn't have bothered if the list wasn't open.) -- Michel Fortin michel.fortin michelf.com http://michelf.com/
Dec 28 2009
parent reply Walter Bright <newshound1 digitalmars.com> writes:
Michel Fortin wrote:
 Also keep in mind that we don't really need a shared vision among 
 everyone. What's needed is someone who takes the decisions. Discussion 
 is only needed to help that person take the right decisions. Although 
 consensus among all members certainly boosts the decider 
 self-confidence, it is not required, and not necessarily desirable 
 either. A consensus among only a few key people is all that is needed, 
 and this has little to do with who is allowed to raise issues and 
 propose solutions.

The real problem with a concurrency model is that very few programmers understand the issues. The failed Java concurrency model is an example of this shortage. For another, about 5 years ago I attended a panel of 30 of the top C++ experts in the world to discuss a concurrency model for C++0x. It didn't take long for it to become obvious that exactly two people understood the issues - Hans Boehm and Herb Sutter. The rest of us sat there slack-jawed and drooling, asking endless inane questions. I wish I had the patience Hans and Herb showed in dealing with this. Since then I have tried to master this topic, but I don't have much experience writing complex multithreaded code. So what we need are people who are experienced with MT code who can evaluate the design to see if we missed the boat or not. I'd rather not shoot at the moon yet wind up orbiting some asteroid.
Jan 08 2010
next sibling parent Jason House <jason.james.house gmail.com> writes:
Walter Bright Wrote:

 Since then I have tried to master this topic, but I don't have much 
 experience writing complex multithreaded code. So what we need are 
 people who are experienced with MT code who can evaluate the design to 
 see if we missed the boat or not. I'd rather not shoot at the moon yet 
 wind up orbiting some asteroid.

I can't comment on the future threading model, but I have been trying to use the current one by converting an existing multithreaded app. I've gotten a lot further than my last update on this list. Bugzilla 3660 is quite severe. My current understanding of the issue is that matching of function templates to attempted function calls isn't aware of the distinction between shared and non-shared functions.
Jan 08 2010
prev sibling next sibling parent reply Brien <reply tolist.com> writes:
Walter Bright Wrote:
 Since then I have tried to master this topic, but I don't have much 
 experience writing complex multithreaded code. So what we need are 
 people who are experienced with MT code who can evaluate the design to 
 see if we missed the boat or not. I'd rather not shoot at the moon yet 
 wind up orbiting some asteroid.

Walter, How would you characterize D1.0 in terms of it's use for big multithreaded applications? Does the compiler produce safe/performant code? Thanks, Brien
Jan 08 2010
parent Walter Bright <newshound1 digitalmars.com> writes:
Brien wrote:
 How would you characterize D1.0 in terms of it's use for big
 multithreaded applications?   Does the compiler produce
 safe/performant code?

D 1.0 is about where Java is in its support for MT programming. You can do it, but you get little or no support from the type system.
Jan 08 2010
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
retard wrote:
 Fri, 08 Jan 2010 00:50:26 -0800, Walter Bright wrote:
 
 Michel Fortin wrote:
 Also keep in mind that we don't really need a shared vision among
 everyone. What's needed is someone who takes the decisions. Discussion
 is only needed to help that person take the right decisions. Although
 consensus among all members certainly boosts the decider
 self-confidence, it is not required, and not necessarily desirable
 either. A consensus among only a few key people is all that is needed,
 and this has little to do with who is allowed to raise issues and
 propose solutions.

understand the issues. The failed Java concurrency model is an example of this shortage. For another, about 5 years ago I attended a panel of 30 of the top C++ experts in the world to discuss a concurrency model for C++0x. It didn't take long for it to become obvious that exactly two people understood the issues - Hans Boehm and Herb Sutter. The rest of us sat there slack-jawed and drooling, asking endless inane questions. I wish I had the patience Hans and Herb showed in dealing with this. Since then I have tried to master this topic, but I don't have much experience writing complex multithreaded code. So what we need are people who are experienced with MT code who can evaluate the design to see if we missed the boat or not. I'd rather not shoot at the moon yet wind up orbiting some asteroid.

If only 2 of the top 30 c++ experts understand concurrency models, what makes you think that D programming community, mostly consisting of novice/ hobbyist/student programmers have any better luck? There are only a handful of such experts as Sean, Andrei, Bartosz, Brad et al here. The D community is maybe 3 to 4 orders of magnitude smaller than the global C++ community. Only a handful of enterprises use D and there is basically no academic research happening around D.

Our flagship for communicating across concurrent entities is message passing, which is much more palatable than classic share-the-world concurrency models. I'm very excited about how this is coming together. Andrei
Jan 08 2010
prev sibling next sibling parent Walter Bright <newshound1 digitalmars.com> writes:
Steven E. Harris wrote:
 Walter Bright <newshound1 digitalmars.com> writes:
 
 The failed Java concurrency model is an example of this shortage.

Can you elaborate? Are you talking pre-JSR-000133¹? Footnotes: ¹ http://jcp.org/aboutJava/communityprocess/review/jsr133/index.html

In Java, you can share data between threads without restriction. Hence, code is vulnerable to all the sequential consistency bugs.
Jan 08 2010
prev sibling next sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
retard wrote:
 If only 2 of the top 30 c++ experts understand concurrency models, what 
 makes you think that D programming community, mostly consisting of novice/
 hobbyist/student programmers have any better luck?

That is why the flagship of doing concurrency in D will be message passing, which is how Erlang, Scala, and Go do concurrency. Using message passing will avoid any need by the user to understand sharing, sequential consistency, and synchronization issues. Sharing and synchronization will be for the advanced programmer.
 Only a handful of enterprises use D and there is basically no 
 academic research happening around D.

That'll change!
Jan 08 2010
parent reply Walter Bright <newshound1 digitalmars.com> writes:
retard wrote:
 My point was, what makes D community members more capable of reasoning 
 about these issues.

Nothing. What makes D special, though, is better language support for sound concurrent programming. But you'll still need a mental model of what's going on, just like you cannot program very successfully in C if you have no mental model of what a pointer is.
 I bet it's obvious to the top 30 C++ experts that 
 other kinds of concurrency models exist than multi-threaded shared memory 
 abstraction or SIMD. The problem isn't just about coming up with a model 
 -- if it helps you in any way, go ahead and browse old academic papers. 
 E.g. CSP was invented 32 years ago.
 
 There are message passing libs for C++, too, for instance MPI. Making the 
 feature built-in doesn't automatically make creating concurrent 
 applications easier.

Yes it does. C++'s (and Java's, Scala's, and Go's) gaping problem is there's no way to ensure that you don't have implicit sharing going on. In D, such can be guaranteed.
 For instance message passing is harder to use than 
 implicit parallelism and auto-vectorization when doing parallel 
 processing in loops.

Sure. Except that implicit parallelism is inherently unsound unless the type system can support it. I'll go further than that. Any language that supports implicit sharing of mutable data is inherently unsound for multithreaded programming.
Jan 08 2010
parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Walter Bright (newshound1 digitalmars.com)'s article
 Sure. Except that implicit parallelism is inherently unsound unless the
 type system can support it.
 I'll go further than that. Any language that supports implicit sharing
 of mutable data is inherently unsound for multithreaded programming.

One thing that I think needs to be considered in the concurrency architecture is the case of performance-critical massive data parallelism. In these cases concurrency is actually not very hard and a simple parallel foreach loop covers a lot of cases. As far as safety, the amount of code being executed inside a parallel foreach loop is generally small enough that it's easy to reason about, and thus it's ok not to have any hard static guarantees and leave safety up to the programmer, as long as the programmer understands at least the basics of concurrency. I would like to see a situation where OpenMP/ParallelFuture-style concurrency is still implementable in D without unreasonable performance or syntactic overhead after the new concurrency system is fully in place.
Jan 08 2010
parent reply Walter Bright <newshound1 digitalmars.com> writes:
dsimcha wrote:
 == Quote from Walter Bright (newshound1 digitalmars.com)'s article
 Sure. Except that implicit parallelism is inherently unsound unless the
 type system can support it.
 I'll go further than that. Any language that supports implicit sharing
 of mutable data is inherently unsound for multithreaded programming.

One thing that I think needs to be considered in the concurrency architecture is the case of performance-critical massive data parallelism. In these cases concurrency is actually not very hard and a simple parallel foreach loop covers a lot of cases. As far as safety, the amount of code being executed inside a parallel foreach loop is generally small enough that it's easy to reason about, and thus it's ok not to have any hard static guarantees and leave safety up to the programmer, as long as the programmer understands at least the basics of concurrency.

It's hard to reason about even small examples. Consider the double checked locking bug. I even thought I understood it, until Bartosz discovered I'd written one into Phobos. As for understanding the basics of concurrency, I defy anyone to read the Intel CPU manuals and discover how memory read/write barriers work on Intel chips and how and when they should be used. With C++, everyone using memory barriers has to figure it out, and the consequences of getting it wrong are very, very, very obscure bugs. With D, only the compiler writer has to get it right.
 I would like to see a situation where OpenMP/ParallelFuture-style concurrency
is
 still implementable in D without unreasonable performance or syntactic overhead
 after the new concurrency system is fully in place.

There's no reason it cannot be. At the worst, you can cast away "shared" and cowboy it exactly as you would in C++. There is no circumstance where you are worse off with D than with C++, and quite a lot of cases where you're much better off.
Jan 08 2010
parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Walter Bright (newshound1 digitalmars.com)'s article
 dsimcha wrote:
 == Quote from Walter Bright (newshound1 digitalmars.com)'s article
 Sure. Except that implicit parallelism is inherently unsound unless the
 type system can support it.
 I'll go further than that. Any language that supports implicit sharing
 of mutable data is inherently unsound for multithreaded programming.

One thing that I think needs to be considered in the concurrency architecture is the case of performance-critical massive data parallelism. In these cases concurrency is actually not very hard and a simple parallel foreach loop covers a lot of cases. As far as safety, the amount of code being executed inside a parallel foreach loop is generally small enough that it's easy to reason about, and thus it's ok not to have any hard static guarantees and leave safety up to the programmer, as long as the programmer understands at least the basics of


 It's hard to reason about even small examples. Consider the double
 checked locking bug. I even thought I understood it, until Bartosz
 discovered I'd written one into Phobos.
 As for understanding the basics of concurrency, I defy anyone to read
 the Intel CPU manuals and discover how memory read/write barriers work
 on Intel chips and how and when they should be used.
 With C++, everyone using memory barriers has to figure it out, and the
 consequences of getting it wrong are very, very, very obscure bugs. With
 D, only the compiler writer has to get it right.
 I would like to see a situation where OpenMP/ParallelFuture-style concurrency
is
 still implementable in D without unreasonable performance or syntactic overhead
 after the new concurrency system is fully in place.

At the worst, you can cast away "shared" and cowboy it exactly as you would in C++. There is no circumstance where you are worse off with D than with C++, and quite a lot of cases where you're much better off.

Ok, one thing that leads to a lot of misunderstanding and confusion is that there's (as far as I can tell) absolutely no documentation of what shared does/will do or what its semantics are/will be, other than a few newsgroup posts from years ago,which were tentative, and the "migrating to shared" article, which is mostly about thread-local storage. Also, as far as I can tell, shared is not fully implemented yet, because core.thread doesn't require a shared delegate. Is this correct? Here are some questions, and this is just what I've been able to think of off the top of my head in a few minutes: What guarantees is shared supposed to provide? What does shared have to do with synchronization? What does shared have to do with memory barriers? What are the semantics of casting FROM unshared TO shared? What are the semantics of casting FROM shared TO unshared? Why doesn't shared appear anywhere in core.thread?
Jan 08 2010
next sibling parent reply Walter Bright <newshound1 digitalmars.com> writes:
dsimcha wrote:
 Ok, one thing that leads to a lot of misunderstanding and confusion is that
 there's (as far as I can tell) absolutely no documentation of what shared
 does/will do or what its semantics are/will be, other than a few newsgroup
posts
 from years ago,which were tentative, and the "migrating to shared" article,
which
 is mostly about thread-local storage.  Also, as far as I can tell, shared is
not
 fully implemented yet, because core.thread doesn't require a shared delegate. 
Is
 this correct?

Yes.
 Here are some questions, and this is just what I've been able to think of off
the
 top of my head in a few minutes:
 
 What guarantees is shared supposed to provide?

Shared means that multiple threads can access the data. The guarantee is that if it is not shared, and not immutable, that only the current thread can see it.
 What does shared have to do with synchronization?

Only shared data can be synchronized. It makes no sense to synchronize thread local data.
 What does shared have to do with memory barriers?

Reading/writing shared data emits memory barriers to ensure sequential consistency (not implemented).
 What are the semantics of casting FROM unshared TO shared?

You'd better be sure there are no other unshared references to that same data.
 What are the semantics of casting FROM shared TO unshared?

You'd better be sure there are no other shared references to that same data.
 Why doesn't shared appear anywhere in core.thread?

Because nobody has spent the effort to do it.
Jan 08 2010
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Walter Bright wrote:
 dsimcha wrote:
 Ok, one thing that leads to a lot of misunderstanding and confusion is 
 that
 there's (as far as I can tell) absolutely no documentation of what shared
 does/will do or what its semantics are/will be, other than a few 
 newsgroup posts
 from years ago,which were tentative, and the "migrating to shared" 
 article, which
 is mostly about thread-local storage.  Also, as far as I can tell, 
 shared is not
 fully implemented yet, because core.thread doesn't require a shared 
 delegate.  Is
 this correct?

Yes.
 Here are some questions, and this is just what I've been able to think 
 of off the
 top of my head in a few minutes:

 What guarantees is shared supposed to provide?

Shared means that multiple threads can access the data. The guarantee is that if it is not shared, and not immutable, that only the current thread can see it.
 What does shared have to do with synchronization?

Only shared data can be synchronized. It makes no sense to synchronize thread local data.
 What does shared have to do with memory barriers?

Reading/writing shared data emits memory barriers to ensure sequential consistency (not implemented).
 What are the semantics of casting FROM unshared TO shared?

You'd better be sure there are no other unshared references to that same data.
 What are the semantics of casting FROM shared TO unshared?

You'd better be sure there are no other shared references to that same data.
 Why doesn't shared appear anywhere in core.thread?

Because nobody has spent the effort to do it.

Thanks Walter for doing this. These questions and their answers could start an FAQ page on D's website. Andrei
Jan 09 2010
parent Jesse Phillips <jessekphillips+D gmail.com> writes:
Andrei Alexandrescu wrote:

 Thanks Walter for doing this. These questions and their answers could 
 start an FAQ page on D's website.

 Andrei

In the mean time: http://www.prowiki.org/wiki4d/wiki.cgi?DevelopmentWithD/FAQ/Shared
Jan 09 2010
prev sibling parent Sean Kelly <sean invisibleduck.org> writes:
dsimcha Wrote:
 Why doesn't shared appear anywhere in core.thread?

Until the semantics of shared are finalized and the critical bugs are worked out I don't want to integrate shared with core.thread--it's the centerpoint of too much stuff (GC, etc). But general purpose concurrency should never touch core.thread anyway. There will be a higher-level interface available that's safer and easier to use. One could also argue that core.thread should accept unshared delegates, since it's possible to do so safely, you just don't have any language support to help. The design philosophy with the modules in core is quite different from std.
Jan 08 2010
prev sibling parent Michel Fortin <michel.fortin michelf.com> writes:
On 2010-01-08 03:50:26 -0500, Walter Bright <newshound1 digitalmars.com> said:

 Michel Fortin wrote:
 Also keep in mind that we don't really need a shared vision among 
 everyone. What's needed is someone who takes the decisions. Discussion 
 is only needed to help that person take the right decisions. Although 
 consensus among all members certainly boosts the decider 
 self-confidence, it is not required, and not necessarily desirable 
 either. A consensus among only a few key people is all that is needed, 
 and this has little to do with who is allowed to raise issues and 
 propose solutions.

The real problem with a concurrency model is that very few programmers understand the issues. [...] Since then I have tried to master this topic, but I don't have much experience writing complex multithreaded code. So what we need are people who are experienced with MT code who can evaluate the design to see if we missed the boat or not. I'd rather not shoot at the moon yet wind up orbiting some asteroid.

I'm not sure what you're replying to in the above message. I was simply saying that to accomplish progress in something, the only necessary condition is to have an understanding among those who will actually implement that thing. Other opinions can be very useful, they can also be irrelevant, but you don't absolutely need a consensus among every participant in the discussion for something to advance. You're making a case for making "shared" help programmers because most don't grasp all the implications of shared memory between processors. That's good but I don't really see how it relates. By the way, I have seen my lot of multithreaded code in C++, sometime very well written, sometime just wrong, and sometime correct but obscure enough that takes help from the original author to understand. I've written code that probably enters in all these categories, and debugged my share of hard to track concurrency issues too. Were I was working previously, the most time-consuming bugs were always concurrency issues, generally race conditions happening every few hours/days or in certain rare conditions but not always, causing either deadlocks or other problematic behaviors. Fortunately, we also had failsafes to stop things in case the software would crash as it had safety implications. Although I'm no longer working there and dealing much less with concurrency issues now, I'm still interested in the subject. -- Michel Fortin michel.fortin michelf.com http://michelf.com/
Jan 09 2010
prev sibling next sibling parent retard <re tard.com.invalid> writes:
Mon, 28 Dec 2009 10:20:53 -0600, Andrei Alexandrescu wrote:

 Michel Fortin wrote:
 On 2009-12-27 15:32:52 -0500, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> said:
 
 I think we are now in the position of defining a solid set of
 concurrency primitives for D. This follows many months of mulling over
 models and options.

 It would be great to open the participation to the design as broadly
 as possible, but I think it's realistic to say we won't be able to get
 things done on the newsgroup. When we discuss a topic around here,
 there's plenty of good ideas but also the inevitable bikeshed
 discussions, explanations being asked, explanations being given, and
 other sources of noise. We simply don't have the time to deal with all
 that - the time is short and we only have one shot at this.

 That's why I'm thinking of creating a mailing list or maybe another
 group for this. Any ideas on what would be the best approach? I also
 want to gauge interest from threading experts who'd like to
 participate. Please advise: (a) whether you would like to participate
 to the design; (b) keep discussions on the general group; (c) create a
 separate newsgroup; (d) create a mailing list. The latter would have
 open enrollment.

I think it should be as open as possible. If done in a separate smaller group, it may be a good idea to post reports to the general newsgroup more or less regularly so that those who cannot participate in the detailed discussions have an idea of where it's going, and also to get more general input.

That's obviously the best way to go, but there are a couple of circumstances that make that more difficult. 1. Chapter drafts will be the basis for discussion, but understandably the publisher does not allow me to freely distribute them. 2. Time. There are regulars on this group that have a "when in doubt, make them sweat" policy. I think it's a very good and gainful attitude for everyone involved, and I generally enjoy discussing this or that idea because it helps me and others gain a better understanding, but this time there won't be much time for discussions of the form: a) Poster: "Subtle issue X sounds like a bad idea. I don't agree with it." b) <Long argumentation back and forth.> c) Poster: "I stay unconvinced." or "That makes sense." There will be very little time for anything like this, particularly if explaining X requires a fair amount of background building. Building a shared vision is very difficult among only a small group of people, and doing so for a larger group will be an enormous drag. I feel very lucky that Walter and I share views most of the time (except, of course, when he's wrong :o)).

I think a shared vision is easier to achieve in a small group. If the group size is 1, you only need to convince yourself, if 2, you have only one other person who needs to agree and so forth. In groups like c++ standards committee every player in the software industry wants to be heard. That's why it's not gonna be c++0x anymore, maybe c++1x (c++0x0a) or c++2x. I don't like how in D community new features are added. We often first get a binary dmd release containing the new feature, after that some obscure half-official explanation about the corner cases that will be fixed in the next release. Only after that Walter might answer 'Yes.' to some well laid questions and finally after 2-3 years a very terse informal text is added to the specification. What kind of experts are you exactly seeking now? Guys who build massive and scalable native/green thread pools to serve http requests? Guys who compute weather forecasts with 10.000 node MPI setups? Is it going to be a multi-threaded, multi-core, or multi-computer architecture?
Dec 28 2009
prev sibling next sibling parent Tim Matthews <tim.matthews7 gmail.com> writes:
On 29/12/2009 3:48 a.m., Michel Fortin wrote:
 On 2009-12-27 15:32:52 -0500, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> said:

 I think we are now in the position of defining a solid set of
 concurrency primitives for D. This follows many months of mulling over
 models and options.

 It would be great to open the participation to the design as broadly
 as possible, but I think it's realistic to say we won't be able to get
 things done on the newsgroup. When we discuss a topic around here,
 there's plenty of good ideas but also the inevitable bikeshed
 discussions, explanations being asked, explanations being given, and
 other sources of noise. We simply don't have the time to deal with all
 that - the time is short and we only have one shot at this.

 That's why I'm thinking of creating a mailing list or maybe another
 group for this. Any ideas on what would be the best approach? I also
 want to gauge interest from threading experts who'd like to
 participate. Please advise: (a) whether you would like to participate
 to the design; (b) keep discussions on the general group; (c) create a
 separate newsgroup; (d) create a mailing list. The latter would have
 open enrollment.

I think it should be as open as possible. If done in a separate smaller group, it may be a good idea to post reports to the general newsgroup more or less regularly so that those who cannot participate in the detailed discussions have an idea of where it's going, and also to get more general input. About the bikeshed issue, I'm not sure how much those bikeshed discussions are slowing down the more important ones, but they often start from legitimate real, often syntactic, issues. Those discussions shouldn't be avoided just because everyone has an opinion. But perhaps regular reports to the general newsgroup would help confining them there. I'd be in favor of creating a newsgroup for concurrency, and I'll probably want to participate a little too, although I'm not sure how much yet.

I agree with the open as possible +1. A closed model would likely only permit fully committed and those that have no say without letting anything in between where as the open allows some feedback. As for off topic: This can be avoided by using another form of communication. Maybe just add a few announcements here and put all proposals with for/against arguments on the wiki. Or anything that supports some sort of moderation/editing but I think wiki is perfect for this.
Dec 28 2009
prev sibling next sibling parent "Steven E. Harris" <seh panix.com> writes:
Walter Bright <newshound1 digitalmars.com> writes:

 The failed Java concurrency model is an example of this shortage.

Can you elaborate? Are you talking pre-JSR-000133=C2=B9? Footnotes:=20 =C2=B9 http://jcp.org/aboutJava/communityprocess/review/jsr133/index.html --=20 Steven E. Harris
Jan 08 2010
prev sibling next sibling parent retard <re tard.com.invalid> writes:
Fri, 08 Jan 2010 00:50:26 -0800, Walter Bright wrote:

 Michel Fortin wrote:
 Also keep in mind that we don't really need a shared vision among
 everyone. What's needed is someone who takes the decisions. Discussion
 is only needed to help that person take the right decisions. Although
 consensus among all members certainly boosts the decider
 self-confidence, it is not required, and not necessarily desirable
 either. A consensus among only a few key people is all that is needed,
 and this has little to do with who is allowed to raise issues and
 propose solutions.

The real problem with a concurrency model is that very few programmers understand the issues. The failed Java concurrency model is an example of this shortage. For another, about 5 years ago I attended a panel of 30 of the top C++ experts in the world to discuss a concurrency model for C++0x. It didn't take long for it to become obvious that exactly two people understood the issues - Hans Boehm and Herb Sutter. The rest of us sat there slack-jawed and drooling, asking endless inane questions. I wish I had the patience Hans and Herb showed in dealing with this. Since then I have tried to master this topic, but I don't have much experience writing complex multithreaded code. So what we need are people who are experienced with MT code who can evaluate the design to see if we missed the boat or not. I'd rather not shoot at the moon yet wind up orbiting some asteroid.

If only 2 of the top 30 c++ experts understand concurrency models, what makes you think that D programming community, mostly consisting of novice/ hobbyist/student programmers have any better luck? There are only a handful of such experts as Sean, Andrei, Bartosz, Brad et al here. The D community is maybe 3 to 4 orders of magnitude smaller than the global C++ community. Only a handful of enterprises use D and there is basically no academic research happening around D.
Jan 08 2010
prev sibling next sibling parent retard <re tard.com.invalid> writes:
Fri, 08 Jan 2010 12:33:06 -0800, Walter Bright wrote:

 retard wrote:
 If only 2 of the top 30 c++ experts understand concurrency models, what
 makes you think that D programming community, mostly consisting of
 novice/ hobbyist/student programmers have any better luck?

That is why the flagship of doing concurrency in D will be message passing, which is how Erlang, Scala, and Go do concurrency. Using message passing will avoid any need by the user to understand sharing, sequential consistency, and synchronization issues. Sharing and synchronization will be for the advanced programmer.

My point was, what makes D community members more capable of reasoning about these issues. I bet it's obvious to the top 30 C++ experts that other kinds of concurrency models exist than multi-threaded shared memory abstraction or SIMD. The problem isn't just about coming up with a model -- if it helps you in any way, go ahead and browse old academic papers. E.g. CSP was invented 32 years ago. There are message passing libs for C++, too, for instance MPI. Making the feature built-in doesn't automatically make creating concurrent applications easier. For instance message passing is harder to use than implicit parallelism and auto-vectorization when doing parallel processing in loops.
Jan 08 2010
prev sibling parent retard <re tard.com.invalid> writes:
Fri, 08 Jan 2010 23:12:38 +0000, dsimcha wrote:

 == Quote from Walter Bright (newshound1 digitalmars.com)'s article
 Sure. Except that implicit parallelism is inherently unsound unless the
 type system can support it.
 I'll go further than that. Any language that supports implicit sharing
 of mutable data is inherently unsound for multithreaded programming.

One thing that I think needs to be considered in the concurrency architecture is the case of performance-critical massive data parallelism. In these cases concurrency is actually not very hard and a simple parallel foreach loop covers a lot of cases. As far as safety, the amount of code being executed inside a parallel foreach loop is generally small enough that it's easy to reason about, and thus it's ok not to have any hard static guarantees and leave safety up to the programmer, as long as the programmer understands at least the basics of concurrency. I would like to see a situation where OpenMP/ParallelFuture-style concurrency is still implementable in D without unreasonable performance or syntactic overhead after the new concurrency system is fully in place.

These systems also solve different kinds of problems. MMX/SSE/Altivec and now GPU hardware solves problems where you run a single operation on a large set of data simultaneously. The communication is really cheap because e.g. in SSE data is in "ordinary" registers which are the fastest memory units available. Message passing system scales well if you have many computers that form a computing cluster. It also performs well when the computation can be split into distinct tasks which run in separate threads and communicate rarely. Unless messages are translated into native SIMD primitives in data parallel cases, message passing will be much slower when you do that kind of processing. Parallel loops don't scale that well on multiple computing nodes - the computation needs to be split into larger tasks to minimize communication delays since nodes simply can't share the same thread state.
Jan 08 2010
prev sibling next sibling parent "Phil Deets" <pjdeets2 gmail.com> writes:
On Sun, 27 Dec 2009 15:32:52 -0500, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 That's why I'm thinking of creating a mailing list or maybe another  
 group for this. Any ideas on what would be the best approach? I also  
 want to gauge interest from threading experts who'd like to participate.  
 Please advise: (a) whether you would like to participate to the design;  
 (b) keep discussions on the general group; (c) create a separate  
 newsgroup; (d) create a mailing list. The latter would have open  
 enrollment.

I probably wouldn't contribute much, but I'm satisfied as long as I can view the discussion. I have learned a lot from the D newsgroups. I would strongly prefer a newsgroup to a mailing list. I wish every mailing list on the Internet would convert to a newsgroup. The rest of this message is just a dump of my limited experience with threading in case its useful for you to see the perspective of someone without a lot of threading experience. My only major experience with concurrency involved a program which was split into a GUI (using C#/WPF) and a C++ DLL with three always-on threads plus a thread from C# coming in occasionally using PInvoke. The three threads in the C++ side were in while(true) loops which checked volatile boolean flags for work and slept briefly when there was nothing to do. (See below for a general idea of what the code looked like.) I didn't use any synchronization except the volatile flags and global volatile sets of parameters. This worked for me since only one thread handled any one piece of functionality. So some built-in way of low overhead cross-thread function calling would be nice. Or if there was a better way to do it, a design that would nudge me in the right direction would be good. Actually, after looking at the code below, I'm not convinced I got it right since TakeData could possibly be called before thread1_buffer got filled. globals: volatile bool take_data_flag = false; volatile char* take_data_buffer = null; // constant buffer size (something like 32KiB) thread1: void TakeData(char* buffer) { take_data_buffer = buffer; take_data_flag = true; take_data_called = true; } bool take_data_called = false; char thread1_buffer[BUFFER_SIZE]; while (true) { if (!take_data_called) { // Add data to thread1_buffer if (<buffer is full>) { TakeData(thread1_buffer); } } else { if (take_data_flag == false) { // ok to use buffer again take_data_called = false; } } sleep(50); } thread2: char thread2_buffer[BUFFER_SIZE]; while (true) { if (take_data_flag) { // copy take_data_buffer to thread2_buffer take_data_flag = false; // process thread2_buffer } else sleep(50); }
Dec 28 2009
prev sibling next sibling parent Leandro Lucarella <llucax gmail.com> writes:
Phil Deets, el 28 de diciembre a las 11:09 me escribiste:
 On Sun, 27 Dec 2009 15:32:52 -0500, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:
 
That's why I'm thinking of creating a mailing list or maybe
another group for this. Any ideas on what would be the best
approach? I also want to gauge interest from threading experts
who'd like to participate. Please advise: (a) whether you would
like to participate to the design; (b) keep discussions on the
general group; (c) create a separate newsgroup; (d) create a
mailing list. The latter would have open enrollment.

I probably wouldn't contribute much, but I'm satisfied as long as I can view the discussion. I have learned a lot from the D newsgroups. I would strongly prefer a newsgroup to a mailing list. I wish every mailing list on the Internet would convert to a newsgroup.

Then you will looove Gmane, because it is exactly that a NNTP gateway for almost any ML: http://www.gmane.org/ (you can add missing mailing lists) BTW, I think all the discussion should be here, if there is no time to discuss, there is no point in creating another ML/NG, just discuss it privately and be done with it (I really think it would be a very bad idea though). -- Leandro Lucarella (AKA luca) http://llucax.com.ar/ ---------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ---------------------------------------------------------------------- "Somos testigos de Jaimito, venimos a traer la salvación, el mundo va a desaparecer, somos testigos de Jaimito!". Nos enyoguizamos... Así que "somos testigos"? Te dejo el culo hecho un vino, y la conch'itumá, y la conch'itumá! -- Sidharta Kiwi
Dec 28 2009
prev sibling next sibling parent "Phil Deets" <pjdeets2 gmail.com> writes:
On Mon, 28 Dec 2009 12:46:25 -0500, Leandro Lucarella <llucax gmail.com>  
wrote:

 Then you will looove Gmane, because it is exactly that a NNTP gateway for
 almost any ML:
 http://www.gmane.org/

 (you can add missing mailing lists)

Thanks for the link. It looks awesome.
Dec 28 2009
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Sun, 27 Dec 2009 12:32:52 -0800, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 I think we are now in the position of defining a solid set of  
 concurrency primitives for D. This follows many months of mulling over  
 models and options.

 It would be great to open the participation to the design as broadly as  
 possible, but I think it's realistic to say we won't be able to get  
 things done on the newsgroup. When we discuss a topic around here,  
 there's plenty of good ideas but also the inevitable bikeshed  
 discussions, explanations being asked, explanations being given, and  
 other sources of noise. We simply don't have the time to deal with all  
 that - the time is short and we only have one shot at this.

 That's why I'm thinking of creating a mailing list or maybe another  
 group for this. Any ideas on what would be the best approach? I also  
 want to gauge interest from threading experts who'd like to participate.  
 Please advise: (a) whether you would like to participate to the design;  
 (b) keep discussions on the general group; (c) create a separate  
 newsgroup; (d) create a mailing list. The latter would have open  
 enrollment.


 Andrei

I'd like to participate and prefer newsgroups to mailing lists. I think a separate newsgroup, although some work to setup, would make it easier to follow the discussions.
Dec 29 2009
prev sibling next sibling parent reply Nick B <"nick_NOSPAM_.barbalich" gmail.com> writes:
Andrei Alexandrescu wrote:
 I think we are now in the position of defining a solid set of 
 concurrency primitives for D. This follows many months of mulling over 
 models and options.
 

 
 That's why I'm thinking of creating a mailing list or maybe another 
 group for this. Any ideas on what would be the best approach? I also 
 want to gauge interest from threading experts who'd like to participate. 
 Please advise: (a) whether you would like to participate to the design; 
 (b) keep discussions on the general group; (c) create a separate 
 newsgroup; (d) create a mailing list. The latter would have open 
 enrollment.
 
 
 Andrei

Andrei Will you be inviting Bartosz to participate, or have you already has discussions with him ? He has written a number of blogs around this issue. Nick B
Dec 29 2009
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Nick B wrote:
 Andrei Alexandrescu wrote:
 I think we are now in the position of defining a solid set of 
 concurrency primitives for D. This follows many months of mulling over 
 models and options.

 That's why I'm thinking of creating a mailing list or maybe another 
 group for this. Any ideas on what would be the best approach? I also 
 want to gauge interest from threading experts who'd like to 
 participate. Please advise: (a) whether you would like to participate 
 to the design; (b) keep discussions on the general group; (c) create a 
 separate newsgroup; (d) create a mailing list. The latter would have 
 open enrollment.


 Andrei

Andrei Will you be inviting Bartosz to participate, or have you already has discussions with him ? He has written a number of blogs around this issue. Nick B

I'm not sure about Bartosz' and Walter's thoughts. I personally think that past experience suggests that collaboration would be difficult. No less than three attempts to work together on threads ranging from design to implementation have failed, in spite of everybody's best intentions. Andrei
Dec 29 2009
parent reply Nick B <"nick_NOSPAM_.barbalich" gmail.com> writes:
Andrei Alexandrescu wrote:
 Nick B wrote:


 Andrei

 Will you be inviting Bartosz to participate, or have you already has 
 discussions with him ? He has written a number of  blogs around this 
 issue.

 Nick B

I'm not sure about Bartosz' and Walter's thoughts. I personally think that past experience suggests that collaboration would be difficult. No less than three attempts to work together on threads ranging from design to implementation have failed, in spite of everybody's best intentions. Andrei

Andrei Very interesting to see that you and Walter tried three times !! In Bartosz reply on 14 Oct 2009, [subject: Re: Revamped concurrency API], detailing his detachment from the D community, he gave this opinion: " I'm a bit of a perfectionist and it's hard for me to subscribe to the "good enough" philosophy (as long as it's better that C++, it's fine for D)". If the three of you, can't agree on approach and architecture then I suspect, there is little to be said further. All I can ask is: 1. if you could re-read his blog posts on this subject. 2. Can you explain further this comment (see above ref) by Bartosz: " The semantics of "shared". I can live with postponing the implementation of the race-free type system, but not with the compiler inserting barriers around all shared reads and writes, even inside synchronized sections. " 3. Finally, is this a requirement to be finished for your book, or is this a separate issue ? cheers Nick B
Jan 01 2010
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Nick B wrote:
 Andrei Alexandrescu wrote:
 Nick B wrote:


 Andrei

 Will you be inviting Bartosz to participate, or have you already has 
 discussions with him ? He has written a number of  blogs around this 
 issue.

 Nick B

I'm not sure about Bartosz' and Walter's thoughts. I personally think that past experience suggests that collaboration would be difficult. No less than three attempts to work together on threads ranging from design to implementation have failed, in spite of everybody's best intentions. Andrei

Andrei Very interesting to see that you and Walter tried three times !!

Bartosz too. We all tried, and if we failed I can say for sure it wasn't for the lack of good intentions.
 In Bartosz reply on 14 Oct 2009, [subject: Re: Revamped concurrency 
 API], detailing his detachment from the D community, he gave this 
 opinion: " I'm a bit of a perfectionist and it's hard for me to 
 subscribe to the "good enough" philosophy (as long as it's better that 
 C++, it's fine for D)".  If the three of you, can't agree on approach 
 and architecture then I suspect, there is little to be said further.

There's more than one side to each story. My choice of words describing the situation would be quite different, but please allow me to keep that to myself. Let me just emphasize that the "good enough" philosophy and using C++ as the sole yardstick does not characterize D's development.
 All I can ask is:
 
 1.  if you could re-read his blog posts on this subject.

I am familiar with Bartosz's posts and with the work he is building on.
 2.   Can you explain further this comment (see above ref) by Bartosz:
 " The semantics of "shared". I can live with postponing the 
 implementation of the race-free type system, but not with the compiler 
 inserting barriers around all shared reads and writes, even inside 
 synchronized sections. "

A shared object allows synchronized calls. However, shared is deep and the effect of synchronized is shallow, so there is a conflict. What synchronized effects is a sort of "tail-shared". Walter's and my suggestion was to have synchronized _not_ change the type of the object. That way, the type of each field of the object remains shared-qualified inside a synchronized method, which is correct but overly conservative. In theory the compiler emits barriers whenever reading and writing shared fields. However, inside a synchronized method, the compiler can elide barriers because it understands shared is really tail-shared. Yet there will still be more barriers than strictly necessary because e.g. you could pass the address of a field to another function, which doesn't realize the field is lock-protected. Walter and I deemed the situation rare enough to not complicate the language with the likes of "tail-shared".
 3. Finally, is this a requirement to be finished for your book, or is 
 this a separate issue ?

If TDPL comes without addressing concurrency, or if if comes with something, ahem, not-so-good a la Python, we may as well gather our toys and go home. Andrei
Jan 02 2010
next sibling parent reply Jason House <jason.james.house gmail.com> writes:
Andrei Alexandrescu Wrote:

 Nick B wrote:

 3. Finally, is this a requirement to be finished for your book, or is 
 this a separate issue ?

If TDPL comes without addressing concurrency, or if if comes with something, ahem, not-so-good a la Python, we may as well gather our toys and go home.

The way things are going, I can't see things turning out any other way. Everything involving concurrency/shared seems to get pushed off. This thread is a recent example: asking for participants and then having no meaningful follow-up. When I was part of the sausage-making emails you and Walter were nearly silent, even in response to messages such as "can we at least agree to X?"
Jan 02 2010
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Jason House wrote:
 Andrei Alexandrescu Wrote:
 
 Nick B wrote:

 3. Finally, is this a requirement to be finished for your book,
 or is this a separate issue ?

something, ahem, not-so-good a la Python, we may as well gather our toys and go home.

The way things are going, I can't see things turning out any other way. Everything involving concurrency/shared seems to get pushed off.

We already have a compelling message-passing model designed and to a large extent implemented by Sean Kelly, so you can discount that risk.
 This thread is a recent example: asking for participants and then
 having no meaningful follow-up. When I was part of the sausage-making
 emails you and Walter were nearly silent, even in response to
 messages such as "can we at least agree to X?"

There are a few clerical issue to take care of, and I'm not sure what Walter's preferred method of communication is. I will make you part of whatever transport we'll use. I am sure that some questions weren't answered simply because there was no clear answer. Besides, email communication is imperfect. Andrei
Jan 02 2010
prev sibling parent Nick B <"nick_NOSPAM_.barbalich" gmail.com> writes:
Andrei Alexandrescu wrote:

 
 In Bartosz reply on 14 Oct 2009, [subject: Re: Revamped concurrency 
 API], detailing his detachment from the D community, he gave this 
 opinion: " I'm a bit of a perfectionist and it's hard for me to 
 subscribe to the "good enough" philosophy (as long as it's better that 
 C++, it's fine for D)".  If the three of you, can't agree on approach 
 and architecture then I suspect, there is little to be said further.

There's more than one side to each story. My choice of words describing the situation would be quite different, but please allow me to keep that to myself. Let me just emphasize that the "good enough" philosophy and using C++ as the sole yardstick does not characterize D's development.

[snip]
 
 2.   Can you explain further this comment (see above ref) by Bartosz:
 " The semantics of "shared". I can live with postponing the 
 implementation of the race-free type system, but not with the compiler 
 inserting barriers around all shared reads and writes, even inside 
 synchronized sections. "

A shared object allows synchronized calls. However, shared is deep and the effect of synchronized is shallow, so there is a conflict. What synchronized effects is a sort of "tail-shared". Walter's and my suggestion was to have synchronized _not_ change the type of the object. That way, the type of each field of the object remains shared-qualified inside a synchronized method, which is correct but overly conservative. In theory the compiler emits barriers whenever reading and writing shared fields. However, inside a synchronized method, the compiler can elide barriers because it understands shared is really tail-shared. Yet there will still be more barriers than strictly necessary because e.g. you could pass the address of a field to another function, which doesn't realize the field is lock-protected. Walter and I deemed the situation rare enough to not complicate the language with the likes of "tail-shared".

Thank you for this clarification.
 3. Finally, is this a requirement to be finished for your book, or is 
 this a separate issue ?

If TDPL comes without addressing concurrency, or if if comes with something, ahem, not-so-good a la Python, we may as well gather our toys and go home.

I am sure you and Walter will deliver something good and robust. Nick B
Jan 02 2010
prev sibling next sibling parent "Steven E. Harris" <seh panix.com> writes:
Leandro Lucarella <llucax gmail.com> writes:

 Then you will looove Gmane, because it is exactly that a NNTP gateway for
 almost any ML

As an example, this mailing list is available there as group gmane.comp.lang.d.general However, the mailing list doesn't seem to accept posts through Gmane, despite the reported status of "posting allowed" here: http://dir.gmane.org/gmane.comp.lang.d.general -- Steven E. Harris
Dec 30 2009
prev sibling parent reply Graham St Jack <Graham.StJack internode.on.net> writes:
On Sun, 27 Dec 2009 14:32:52 -0600, Andrei Alexandrescu wrote:

 I think we are now in the position of defining a solid set of
 concurrency primitives for D. This follows many months of mulling over
 models and options.
 
 It would be great to open the participation to the design as broadly as
 possible, but I think it's realistic to say we won't be able to get
 things done on the newsgroup. When we discuss a topic around here,
 there's plenty of good ideas but also the inevitable bikeshed
 discussions, explanations being asked, explanations being given, and
 other sources of noise. We simply don't have the time to deal with all
 that - the time is short and we only have one shot at this.
 
 That's why I'm thinking of creating a mailing list or maybe another
 group for this. Any ideas on what would be the best approach? I also
 want to gauge interest from threading experts who'd like to participate.
 Please advise: (a) whether you would like to participate to the design;
 (b) keep discussions on the general group; (c) create a separate
 newsgroup; (d) create a mailing list. The latter would have open
 enrollment.
 
 
 Andrei

I would love to be involved too.
Jan 03 2010
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Graham St Jack wrote:
 On Sun, 27 Dec 2009 14:32:52 -0600, Andrei Alexandrescu wrote:
 
 I think we are now in the position of defining a solid set of
 concurrency primitives for D. This follows many months of mulling over
 models and options.

 It would be great to open the participation to the design as broadly as
 possible, but I think it's realistic to say we won't be able to get
 things done on the newsgroup. When we discuss a topic around here,
 there's plenty of good ideas but also the inevitable bikeshed
 discussions, explanations being asked, explanations being given, and
 other sources of noise. We simply don't have the time to deal with all
 that - the time is short and we only have one shot at this.

 That's why I'm thinking of creating a mailing list or maybe another
 group for this. Any ideas on what would be the best approach? I also
 want to gauge interest from threading experts who'd like to participate.
 Please advise: (a) whether you would like to participate to the design;
 (b) keep discussions on the general group; (c) create a separate
 newsgroup; (d) create a mailing list. The latter would have open
 enrollment.


 Andrei

I would love to be involved too.

Subscribe to dmd-concurrency from http://lists.puremagic.com/. Andrei
Jan 03 2010