www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Revamped concurrency API

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Occasionally people here ask for ways in which they can help D. One 
thing that would be extremely helpful at this point would be to help 
defining and implementing D's new concurrency API. Unfortunately, 
Bartosz has declined to contribute. That leaves Walter, Sean, Don, and 
participants to this group to do it.

I'm sure you know the three of us are overcommitted, and I'm also sure 
many of you are also in that situation. But if we could somehow focus 
all of the energy and good will in this group to one task of 
extraordinary urgency and importance, that would be awesome.

If anyone has ideas, suggestions, and code to help defining a new 
concurrency API for D, please post.


Andrei
Oct 12 2009
next sibling parent reply Jeremie Pelletier <jeremiep gmail.com> writes:
Andrei Alexandrescu wrote:
 Occasionally people here ask for ways in which they can help D. One 
 thing that would be extremely helpful at this point would be to help 
 defining and implementing D's new concurrency API. Unfortunately, 
 Bartosz has declined to contribute. That leaves Walter, Sean, Don, and 
 participants to this group to do it.
 
 I'm sure you know the three of us are overcommitted, and I'm also sure 
 many of you are also in that situation. But if we could somehow focus 
 all of the energy and good will in this group to one task of 
 extraordinary urgency and importance, that would be awesome.
 
 If anyone has ideas, suggestions, and code to help defining a new 
 concurrency API for D, please post.
 
 
 Andrei

Don't fix it to one model, leave room for multiple concurrency models to be used side by side! I would begin by making the 'shared' specs usable, since the rest will be within the runtime and everything will rely on shared. I already made a post on this some time ago: http://www.digitalmars.com/webnews/newsgroups.php?art_group=digitalmars.D&article_id=96161 Jeremie
Oct 12 2009
next sibling parent reply Fawzi Mohamed <fmohamed mac.com> writes:
 If anyone has ideas, suggestions, and code to help defining a new 
 concurrency API for D, please post.


I think that the correct way to handle concurrency is through a library, D is flexible enough, so that a library can be used, and then one can even write special handlers for special cases. In blip (http://dsource.org/project/blip) I actually did exactly that, in D 1.0 for two reasons: 1) I need 64 bits 2) concurrency is complex, and I had a hard time already with bugs in the stable branch, furthermore the possibility of breakage in very sensitive code that I don't want to touch too often by language changes was not appealing to me. That said there is a feature that would help much D 1.0, and that I would really love to have. I know that D 1.0 is frozen and stuff... but I will try to ask all the same... real closures from delegates when you put a "new" in front of them new delegate(ref int x){x+=y} would create a closure (allocating y). I know that D 2.0 has basically that by default (and scope to get the previous behavior), but as I said D 2.0 is not yet an option for me (I want to work on my projects, I think I am already doing enough for the community), so I thought that asking for a feature that does not break existing D 1.0 code and is in some way already implemented in D 2.0 could be worth trying :) The other D 2.0 features are nice, I do like structure constructors/destructors, post blits, template constraints, const..., well shared I am not so sure about, but anyway all of them are not needed for concurrency. Now about the concurrency infrastructure, here I will discuss SMP parallelization, there is also a more coarse grained parallelization that needs another approach (MPI and agent based model, for now I just wrapped mpi and serialization in a way that could be implemented directly on tcp, and allow to easily communicate arbitrary objects). The concurrency has two sides one is the user/programmer side and the other the efficient realization, I will discuss the user level api, as I realized it in Blip, which is optimized for recursive operations that have to be finished (i.e computations to be performed, not to simulate concurrent systems). The idea is to separate each tasks in chunks that are as small as possible, while still being large enough so that the switching time is small wrt. to the computing time. This subdivision typically is not directly dependent on the number of processors ---- Task is a class the represents a task, it has a string name (for debugging purposes) and can be initialized with a delegate, a function, a fiber or a generator (in two flavors). There are some optimizations to make allocation cheaper. a task can spawn subtasks, and it "knows" how many subtasks there are executing. a task is considered finished when the task is complete and all its subtasks are completed you can append operations to be executed after a task has finished executing. you can wait for a task to finish (but try avoiding it, addint the task to the onFinish of the task is much more efficient). a task has a given level, subtasks have level+1, and tasks that cannot have subtasks have a very high level (int.max/2). a new task can be submitted with t.submit() or t.submitYield() submitYield submits the current task and possibly stops the current one, this together with the fact that tasks with higher level are executed before tasks with lower level means that execution is preferentially a depth first reduction which minimizes the number of suspended tasks (I have also task stealing that steals preferentially the tasks with lowest level, but that part is not yet really used). other important operations are delay and resubmitDelayed that allow one to delay a task, for example when you have to do i/o and you use a method like epoll you can delay the current task, add the file handler to the one to control, and execute resubmitDelayed when that handler has new data. executeNow is useful to execute a task synchronously. That is basically the core that one has to know, the library is actually much more rich, but for a typical usage this is enough. With it one can write code like: import blip.parallel.WorkManager; class SubProblem{ void execute(){ if(problem is large){ SubProblem firstHalf=...; Task("subproblem1",&firstHalf.execute).autorelease.submitYield(); SubProblem secondHalf=...; Task("subproblem2",&secondHalf.execute).autorelease.submitYield(); } else { direct solver } } } solveProblem(){ if(problemIsLarge){ SubProblem wholeProblem=...; Task("solveProblem",&wholeProblem.execute).executeNow(default); } else { direct solver } } just to show a very basic divide and conquer approach
Oct 12 2009
parent reply Fawzi Mohamed <fmohamed mac.com> writes:
On 2009-10-13 03:17:13 +0200, "Robert Jacques" <sandford jhu.edu> said:

 On Mon, 12 Oct 2009 18:21:37 -0400, Fawzi Mohamed <fmohamed mac.com> wrote:
 
 If anyone has ideas, suggestions, and code to help defining a new  
 concurrency API for D, please post.


I think that the correct way to handle concurrency is through a library, D is flexible enough, so that a library can be used, and then one can even write special handlers for special cases. In blip (http://dsource.org/project/blip) I actually did exactly that, in D 1.0 for two reasons:

Looks very interesting. ( Corrected link: http://www.dsource.org/projects/blip ) It looks like it's under the Apache 2.0 license? Would you be willing to put it under the boost license, so that it might be considered for Phobos?

if there is interest, yes
Oct 12 2009
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Fawzi Mohamed wrote:
 On 2009-10-13 03:17:13 +0200, "Robert Jacques" <sandford jhu.edu> said:
 
 On Mon, 12 Oct 2009 18:21:37 -0400, Fawzi Mohamed <fmohamed mac.com> 
 wrote:

 If anyone has ideas, suggestions, and code to help defining a new  
 concurrency API for D, please post.


I think that the correct way to handle concurrency is through a library, D is flexible enough, so that a library can be used, and then one can even write special handlers for special cases. In blip (http://dsource.org/project/blip) I actually did exactly that, in D 1.0 for two reasons:

Looks very interesting. ( Corrected link: http://www.dsource.org/projects/blip ) It looks like it's under the Apache 2.0 license? Would you be willing to put it under the boost license, so that it might be considered for Phobos?

if there is interest, yes

Thanks. There definitely is. I'll start looking at the lib soon, and I recommend others to do the same. Andrei
Oct 12 2009
prev sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Mon, 12 Oct 2009 18:21:37 -0400, Fawzi Mohamed <fmohamed mac.com> wrote:

 If anyone has ideas, suggestions, and code to help defining a new  
 concurrency API for D, please post.


I think that the correct way to handle concurrency is through a library, D is flexible enough, so that a library can be used, and then one can even write special handlers for special cases. In blip (http://dsource.org/project/blip) I actually did exactly that, in D 1.0 for two reasons:

Looks very interesting. ( Corrected link: http://www.dsource.org/projects/blip ) It looks like it's under the Apache 2.0 license? Would you be willing to put it under the boost license, so that it might be considered for Phobos?
Oct 12 2009
prev sibling next sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
== Quote from Andrei Alexandrescu (SeeWebsiteForEmail erdani.org)'s article
 Occasionally people here ask for ways in which they can help D. One
 thing that would be extremely helpful at this point would be to help
 defining and implementing D's new concurrency API. Unfortunately,
 Bartosz has declined to contribute. That leaves Walter, Sean, Don, and
 participants to this group to do it.
 I'm sure you know the three of us are overcommitted, and I'm also sure
 many of you are also in that situation. But if we could somehow focus
 all of the energy and good will in this group to one task of
 extraordinary urgency and importance, that would be awesome.
 If anyone has ideas, suggestions, and code to help defining a new
 concurrency API for D, please post.

For what it's worth, I've been experimenting with message-passing for one leg of the API. I decided to copy the Erlang API because it's very simple and easy to build fancier stuff on top of. What I have so far is: void sendmsg(T)( Pid pid, T val ); final void recvmsg(T...)( Pid pid, T ops ); Pid spawn(T)( T fun ); spawn() is pretty limited so far in that it only spawns threads--I'd expect that function to end up with multiple overloads at some point. Also 'ops' in recvmsg are delegates. Typical use would be: // Thread A sendmsg( pid, 5 ); sendmsg( pid, tuple(5) ); // Thread B recvmsg( pid, (int val) { writefln( "got int: %s", val ); } ); recvmsg( pid, (Tuple!(int) val) { writefln( "got tuple: %s", val ); } ); I thought about using predefined types for "receive any" and "abort after timeout" functions to pass to recvmsg. Pattern matching in D is largely limited to type matching at the moment, which is kind of a limitation, so I'd considered having the delegates return a bool indicating whether the value passed was a match or not. Anyway, thought I'd just add this to the thread in case it sparks discussion. I have a working implementation of this around somewhere if anyone is interested.
Oct 12 2009
next sibling parent Sean Kelly <sean invisibleduck.org> writes:
Sean Kelly Wrote:
 
 void sendmsg(T)( Pid pid, T val );
 final void recvmsg(T...)( Pid pid, T ops );

Oops, I should mention that 'pid' shouldn't be in recvmsg. I just have it in there for now for testing purposes.
Oct 12 2009
prev sibling next sibling parent reply MIURA Masahiro <echochamber gmail.com> writes:
Sean Kelly wrote:
 void sendmsg(T)( Pid pid, T val );
 final void recvmsg(T...)( Pid pid, T ops );
 Pid spawn(T)( T fun );
 
 spawn() is pretty limited so far in that it only spawns threads--I'd
 expect that function to end up with multiple overloads at some point.

Interesting. Future spawn() may spawn the thread in a remote CPU that doesn't share the memory. In that case it perhaps helps to limit fun to be pure, and val to be immutable, as in Erlang. D already has pure and transitive const, so the first few steps for the massively-parallel D programs are already completed! :-)
Oct 12 2009
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
MIURA Masahiro wrote:
 Sean Kelly wrote:
 void sendmsg(T)( Pid pid, T val );
 final void recvmsg(T...)( Pid pid, T ops );
 Pid spawn(T)( T fun );

 spawn() is pretty limited so far in that it only spawns threads--I'd
 expect that function to end up with multiple overloads at some point.

Interesting. Future spawn() may spawn the thread in a remote CPU that doesn't share the memory. In that case it perhaps helps to limit fun to be pure, and val to be immutable, as in Erlang. D already has pure and transitive const, so the first few steps for the massively-parallel D programs are already completed! :-)

That's the plan, except that we weren't thinking of creating remote processes, only sending messages across process and machine boundaries. Here's where notions like value vs. reference and deep cloning become very important. Andrei
Oct 12 2009
next sibling parent Sean Kelly <sean invisibleduck.org> writes:
Andrei Alexandrescu Wrote:

 MIURA Masahiro wrote:
 Sean Kelly wrote:
 void sendmsg(T)( Pid pid, T val );
 final void recvmsg(T...)( Pid pid, T ops );
 Pid spawn(T)( T fun );

 spawn() is pretty limited so far in that it only spawns threads--I'd
 expect that function to end up with multiple overloads at some point.

Interesting. Future spawn() may spawn the thread in a remote CPU that doesn't share the memory. In that case it perhaps helps to limit fun to be pure, and val to be immutable, as in Erlang. D already has pure and transitive const, so the first few steps for the massively-parallel D programs are already completed! :-)

That's the plan, except that we weren't thinking of creating remote processes, only sending messages across process and machine boundaries. Here's where notions like value vs. reference and deep cloning become very important.

Creating remote threads (maybe not OS-level processes) is certainly possible, and I wouldn't be surprised if we did that at some point, assuming something like spawn() is added. But this would be largely invisible at an API level. Like Andrei, I'm more interested in exploring what types should be allowed within messages, what language and library features are necessary, etc. Restricting messages to only shallow value types is certainly possible, but it's rather limiting. I've also experimented with ways to wrap a "shared" container in the same sendmsg/recvmsg interface, but haven't decided if it's an idea worth pursuing yet (or whether it can even be done in a sufficiently robust manner). In any case, that seemed like one way to eliminate obvious use of mutexes for accessing shared data.
Oct 12 2009
prev sibling next sibling parent reply Fawzi Mohamed <fmohamed mac.com> writes:
On 2009-10-13 07:45:41 +0200, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:

 MIURA Masahiro wrote:
 Sean Kelly wrote:
 void sendmsg(T)( Pid pid, T val );
 final void recvmsg(T...)( Pid pid, T ops );
 Pid spawn(T)( T fun );
 
 spawn() is pretty limited so far in that it only spawns threads--I'd
 expect that function to end up with multiple overloads at some point.

Interesting. Future spawn() may spawn the thread in a remote CPU that doesn't share the memory. In that case it perhaps helps to limit fun to be pure, and val to be immutable, as in Erlang. D already has pure and transitive const, so the first few steps for the massively-parallel D programs are already completed! :-)

That's the plan, except that we weren't thinking of creating remote processes, only sending messages across process and machine boundaries. Here's where notions like value vs. reference and deep cloning become very important. Andrei

I think that there are different parallelization levels, and different strategies, there isn't ne that "rules them all". I am very suspicious of systems where data is moved around "magically", normally the cost of that cannot be ignored, so for coarse level parallelization the message passing approach with explicit data distribution done by the programmer is the way to go. On Numa, or really well connected machines having some kind of shared memory is an option, this works well especially if the data is immutable. Finally when one really has shared memory one can go to task scheduling without thinking too much about transfer of memory, it is this last thing that I did address in my previous post. I think that threads are the wrong approach to parallelization at that level, so what I did was to create Tasks that one can use to express more complex relationships between them, so that one can avoids locks almost always, which is both more efficient and less error prone. Fawzi
Oct 12 2009
parent reply Jeremie Pelletier <jeremiep gmail.com> writes:
Fawzi Mohamed wrote:
 On 2009-10-13 07:45:41 +0200, Andrei Alexandrescu 
 <SeeWebsiteForEmail erdani.org> said:
 
 MIURA Masahiro wrote:
 Sean Kelly wrote:
 void sendmsg(T)( Pid pid, T val );
 final void recvmsg(T...)( Pid pid, T ops );
 Pid spawn(T)( T fun );

 spawn() is pretty limited so far in that it only spawns threads--I'd
 expect that function to end up with multiple overloads at some point.

Interesting. Future spawn() may spawn the thread in a remote CPU that doesn't share the memory. In that case it perhaps helps to limit fun to be pure, and val to be immutable, as in Erlang. D already has pure and transitive const, so the first few steps for the massively-parallel D programs are already completed! :-)

That's the plan, except that we weren't thinking of creating remote processes, only sending messages across process and machine boundaries. Here's where notions like value vs. reference and deep cloning become very important. Andrei

I think that there are different parallelization levels, and different strategies, there isn't ne that "rules them all". I am very suspicious of systems where data is moved around "magically", normally the cost of that cannot be ignored, so for coarse level parallelization the message passing approach with explicit data distribution done by the programmer is the way to go. On Numa, or really well connected machines having some kind of shared memory is an option, this works well especially if the data is immutable. Finally when one really has shared memory one can go to task scheduling without thinking too much about transfer of memory, it is this last thing that I did address in my previous post. I think that threads are the wrong approach to parallelization at that level, so what I did was to create Tasks that one can use to express more complex relationships between them, so that one can avoids locks almost always, which is both more efficient and less error prone.

I also don't believe one model is "ruling them all". I agree that threads aren't the best approach, even things like async I/O can be done with futures, other operations can be done with async methods, the actor model is perfect for lots of independent operations that can be executed in any order, message passing is great for event driven designs such as GUIs, software transactional memory works great with state management, and those are only a few models. Threads only need a platform abstraction in the library, from which all other concurrency models can be built. Jeremie
Oct 13 2009
parent reply MIURA Masahiro <echochamber gmail.com> writes:
Jeremie Pelletier wrote:
 I also don't believe one model is "ruling them all".

Let me clarity this, just in case I have caused an unnecessary confusion: I think Sean's Erlang-like API is meant to coexist with the current core.thread, and that you can use one or both of them to build higher-level concurrency models. I think it's nice to have core.thread *and* message-passing API in Phobos. Thread alone is too primitive for daily use, but we don't want to have too many concurrency models.
Oct 13 2009
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
MIURA Masahiro wrote:
 Jeremie Pelletier wrote:
 I also don't believe one model is "ruling them all".

Let me clarity this, just in case I have caused an unnecessary confusion: I think Sean's Erlang-like API is meant to coexist with the current core.thread, and that you can use one or both of them to build higher-level concurrency models. I think it's nice to have core.thread *and* message-passing API in Phobos. Thread alone is too primitive for daily use, but we don't want to have too many concurrency models.

Absolutely! I agree. We need to attack the beast with everything we have. Andrei
Oct 13 2009
next sibling parent Jeremie Pelletier <jeremiep gmail.com> writes:
Robert Jacques wrote:
 On Tue, 13 Oct 2009 11:19:30 -0400, Andrei Alexandrescu 
 <SeeWebsiteForEmail erdani.org> wrote:
 MIURA Masahiro wrote:
 Jeremie Pelletier wrote:
 I also don't believe one model is "ruling them all".

Let me clarity this, just in case I have caused an unnecessary confusion: I think Sean's Erlang-like API is meant to coexist with the current core.thread, and that you can use one or both of them to build higher-level concurrency models. I think it's nice to have core.thread *and* message-passing API in Phobos. Thread alone is too primitive for daily use, but we don't want to have too many concurrency models.

Absolutely! I agree. We need to attack the beast with everything we have. Andrei

I'd recommend reading Bartosz's blog on thread objects vs spawn (http://bartoszmilewski.wordpress.com/2009/09/01/spawning-a thread-the-d-way/). It makes a really good case for why thread objects should never be sub-classed and therefore should be a private, hidden implementation detail and not a public API.

Threads should NOT be private, I don't mind making them final classes but some programs may not require, or some programmers may not want, the high level constructs. If you put the thread construct in core.thread and the concurrent implementations in std.concurrent you still need the std api to access the core threads. But threads being a core module makes it a "here it is should you want it, but check these standard modules first" kind of module. Threads being public also allow any programmer to build custom concurrent models if they're not in the library. Jeremie
Oct 13 2009
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Robert Jacques wrote:
 On Tue, 13 Oct 2009 11:19:30 -0400, Andrei Alexandrescu 
 <SeeWebsiteForEmail erdani.org> wrote:
 MIURA Masahiro wrote:
 Jeremie Pelletier wrote:
 I also don't believe one model is "ruling them all".

Let me clarity this, just in case I have caused an unnecessary confusion: I think Sean's Erlang-like API is meant to coexist with the current core.thread, and that you can use one or both of them to build higher-level concurrency models. I think it's nice to have core.thread *and* message-passing API in Phobos. Thread alone is too primitive for daily use, but we don't want to have too many concurrency models.

Absolutely! I agree. We need to attack the beast with everything we have. Andrei

I'd recommend reading Bartosz's blog on thread objects vs spawn (http://bartoszmilewski.wordpress.com/2009/09/01/spawning-a thread-the-d-way/). It makes a really good case for why thread objects should never be sub-classed and therefore should be a private, hidden implementation detail and not a public API.

I reviewed that article so I guess I read it :o). Bartosz's position is held by many people in the C++ threading community. Andrei
Oct 13 2009
prev sibling parent Jeremie Pelletier <jeremiep gmail.com> writes:
Andrei Alexandrescu wrote:
 MIURA Masahiro wrote:
 Sean Kelly wrote:
 void sendmsg(T)( Pid pid, T val );
 final void recvmsg(T...)( Pid pid, T ops );
 Pid spawn(T)( T fun );

 spawn() is pretty limited so far in that it only spawns threads--I'd
 expect that function to end up with multiple overloads at some point.

doesn't share the memory. In that case it perhaps helps to limit fun to be pure, and val to be immutable, as in Erlang. D already has pure and transitive const, so the first few steps for the massively-parallel D programs are already completed! :-)

That's the plan, except that we weren't thinking of creating remote processes, only sending messages across process and machine boundaries. Here's where notions like value vs. reference and deep cloning become very important. Andrei

With the popularity of cloud computing today and applications pushing toward a client frontend to a server backend, it would only be logical to access server threads from clients. I can see many uses of such a model, for example a cloud IDE that reads from and saves to a specialized version control server, showing what other devs are working on in realtime and being notified instantly of their changes, just like google docs do. That's why I keep saying we need multiple concurrent models in D, there are many usages to every single one of them, and if they can all live together its even better. Jeremie
Oct 13 2009
prev sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 13 Oct 2009 11:19:30 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:
 MIURA Masahiro wrote:
 Jeremie Pelletier wrote:
 I also don't believe one model is "ruling them all".

Let me clarity this, just in case I have caused an unnecessary confusion: I think Sean's Erlang-like API is meant to coexist with the current core.thread, and that you can use one or both of them to build higher-level concurrency models. I think it's nice to have core.thread *and* message-passing API in Phobos. Thread alone is too primitive for daily use, but we don't want to have too many concurrency models.

Absolutely! I agree. We need to attack the beast with everything we have. Andrei

I'd recommend reading Bartosz's blog on thread objects vs spawn (http://bartoszmilewski.wordpress.com/2009/09/01/spawning-a-thread-the-d-way/). It makes a really good case for why thread objects should never be sub-classed and therefore should be a private, hidden implementation detail and not a public API.
Oct 13 2009
prev sibling next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:

Unfortunately, Bartosz has declined to contribute.<

I have read a good amount of his interesting blog posts, he has shown me many things I didn't know about. And he was very happy about the idea of helping D concurrency. Do you know why he has changed his mind regarding this? Maybe because Walter has refused the uniqueness/lend ideas? Bear hugs, bearophile
Oct 12 2009
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
bearophile wrote:
 Andrei Alexandrescu:
 
 Unfortunately, Bartosz has declined to contribute.<

I have read a good amount of his interesting blog posts, he has shown me many things I didn't know about. And he was very happy about the idea of helping D concurrency. Do you know why he has changed his mind regarding this? Maybe because Walter has refused the uniqueness/lend ideas?

You may want to email Bartosz and ask him. Andrei
Oct 12 2009
parent reply Nick B <nickB gmail.com> writes:
Andrei Alexandrescu wrote:
 bearophile wrote:
 Andrei Alexandrescu:

 Unfortunately, Bartosz has declined to contribute.<

I have read a good amount of his interesting blog posts, he has shown me many things I didn't know about. And he was very happy about the idea of helping D concurrency. Do you know why he has changed his mind regarding this? Maybe because Walter has refused the uniqueness/lend ideas?

You may want to email Bartosz and ask him. Andrei

I will ask him, via his blog, and then will post the link. Nick B.
Oct 12 2009
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Nick B wrote:
 Andrei Alexandrescu wrote:
 bearophile wrote:
 Andrei Alexandrescu:

 Unfortunately, Bartosz has declined to contribute.<

I have read a good amount of his interesting blog posts, he has shown me many things I didn't know about. And he was very happy about the idea of helping D concurrency. Do you know why he has changed his mind regarding this? Maybe because Walter has refused the uniqueness/lend ideas?

You may want to email Bartosz and ask him. Andrei

I will ask him, via his blog, and then will post the link.

Great, in the meantime I've also let him know. FWIW, there's no falling out or other juicy details. Bartosz and I are still good friends, we review each other's work, and I eat the mean brioche he bakes whenever I'm given a chance. Andrei
Oct 12 2009
prev sibling parent Nick B <nickB gmail.com> writes:
Nick B wrote:
 Andrei Alexandrescu wrote:
 bearophile wrote:
 Andrei Alexandrescu:

 Unfortunately, Bartosz has declined to contribute.<

I have read a good amount of his interesting blog posts, he has shown me many things I didn't know about. And he was very happy about the idea of helping D concurrency. Do you know why he has changed his mind regarding this? Maybe because Walter has refused the uniqueness/lend ideas?

You may want to email Bartosz and ask him. Andrei

I will ask him, via his blog, and then will post the link. Nick B.

here it is. http://bartoszmilewski.wordpress.com/2009/09/22/ownership-systems-against-data-races/#comment-922 Nick B.
Oct 12 2009
prev sibling parent Nick B <nickB gmail.com> writes:
bearophile wrote:
 Andrei Alexandrescu:
 
 Unfortunately, Bartosz has declined to contribute.<

I have read a good amount of his interesting blog posts, he has shown me many things I didn't know about. And he was very happy about the idea of helping D concurrency. Do you know why he has changed his mind regarding this? Maybe because Walter has refused the uniqueness/lend ideas?

I believe that Bearophile is likely to be on target here. Bartosz has spent a lot of time and effort educating the D community on the subject of data races etc, and many thanks for him for doing this. And now he 'declined to contribute' ? Too busy at work, perhaps ? More likely, a difference in opinion, as to the direction this should proceed, without any discussion on various/conflicting proposals with the community. Is this the way serious architecture decisions should be made ? Would Walter/Bartosz/Andrei like to comment further .... Nick B
Oct 12 2009
prev sibling next sibling parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2009-10-12 11:45:55 -0400, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:

 Occasionally people here ask for ways in which they can help D. One 
 thing that would be extremely helpful at this point would be to help 
 defining and implementing D's new concurrency API. Unfortunately, 
 Bartosz has declined to contribute. That leaves Walter, Sean, Don, and 
 participants to this group to do it.

So Bartosz quit? Not that much surprising given the all too many missing building blocks. I recently realized that you can easily implement unique in a library. It can work pretty much the same as auto_ptr in C++. But to guaranty uniqueness you need either to rely on convention (as in C++), or add support for lent to the compiler. I know we discussed at length how lent could be implemented, we called it 'scope' and 'escape analysis' at the time and the conclusion you (and Walter I guess) came with was that it was too much for D2. Well, without lent (and thus without unique), safe message passing systems across threads will be limited to immutable data, or copied data, which not at all useful in many situations. Now if you want a good concurrency API relying on convention, then that's great: there are plenty of examples to follow out there. But if you want it to be both safe (enforced) and useful at the same time, that's a mission impossible with the current set of tools available from the compiler. That's why I'm not surprised Bartosz declined to implement it. (And sorry if I sound pessimistic.) -- Michel Fortin michel.fortin michelf.com http://michelf.com/
Oct 12 2009
next sibling parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Mon, 12 Oct 2009 21:45:20 -0400, Michel Fortin  
<michel.fortin michelf.com> wrote:

 On 2009-10-12 11:45:55 -0400, Andrei Alexandrescu  
 <SeeWebsiteForEmail erdani.org> said:

 Occasionally people here ask for ways in which they can help D. One  
 thing that would be extremely helpful at this point would be to help  
 defining and implementing D's new concurrency API. Unfortunately,  
 Bartosz has declined to contribute. That leaves Walter, Sean, Don, and  
 participants to this group to do it.

So Bartosz quit? Not that much surprising given the all too many missing building blocks. I recently realized that you can easily implement unique in a library. It can work pretty much the same as auto_ptr in C++. But to guaranty uniqueness you need either to rely on convention (as in C++), or add support for lent to the compiler. I know we discussed at length how lent could be implemented, we called it 'scope' and 'escape analysis' at the time and the conclusion you (and Walter I guess) came with was that it was too much for D2. Well, without lent (and thus without unique), safe message passing systems across threads will be limited to immutable data, or copied data, which not at all useful in many situations. Now if you want a good concurrency API relying on convention, then that's great: there are plenty of examples to follow out there. But if you want it to be both safe (enforced) and useful at the same time, that's a mission impossible with the current set of tools available from the compiler. That's why I'm not surprised Bartosz declined to implement it. (And sorry if I sound pessimistic.)

I agree. Particularly about lent. Immutable and mutable weren't considered complete without const, so I'm surprised that local and shared are considered complete without lent. You can even implement it without escape analysis. Strangely, from what I remember of Bartosz's posts, it was unique that was the sticking point, though you can implement both unique and owned as library types (though they do become less efficient).
Oct 12 2009
parent reply Jeremie Pelletier <jeremiep gmail.com> writes:
Robert Jacques wrote:
 On Mon, 12 Oct 2009 21:45:20 -0400, Michel Fortin 
 <michel.fortin michelf.com> wrote:
 
 On 2009-10-12 11:45:55 -0400, Andrei Alexandrescu 
 <SeeWebsiteForEmail erdani.org> said:

 Occasionally people here ask for ways in which they can help D. One 
 thing that would be extremely helpful at this point would be to help 
 defining and implementing D's new concurrency API. Unfortunately, 
 Bartosz has declined to contribute. That leaves Walter, Sean, Don, 
 and participants to this group to do it.

So Bartosz quit? Not that much surprising given the all too many missing building blocks. I recently realized that you can easily implement unique in a library. It can work pretty much the same as auto_ptr in C++. But to guaranty uniqueness you need either to rely on convention (as in C++), or add support for lent to the compiler. I know we discussed at length how lent could be implemented, we called it 'scope' and 'escape analysis' at the time and the conclusion you (and Walter I guess) came with was that it was too much for D2. Well, without lent (and thus without unique), safe message passing systems across threads will be limited to immutable data, or copied data, which not at all useful in many situations. Now if you want a good concurrency API relying on convention, then that's great: there are plenty of examples to follow out there. But if you want it to be both safe (enforced) and useful at the same time, that's a mission impossible with the current set of tools available from the compiler. That's why I'm not surprised Bartosz declined to implement it. (And sorry if I sound pessimistic.)

I agree. Particularly about lent. Immutable and mutable weren't considered complete without const, so I'm surprised that local and shared are considered complete without lent. You can even implement it without escape analysis. Strangely, from what I remember of Bartosz's posts, it was unique that was the sticking point, though you can implement both unique and owned as library types (though they do become less efficient).

I may sound ignorant, but that does lent means? I mean the word itself, I couldn't find something relevant on google. Anyways, I agree that we need a type qualifier between unshared (local) and shared, just like const sits between mutable and immutable. It would make shared handling just so much easier and convenient. If shared/lent were properly implemented and as easy to use as immutable/const, we'd already begin to see concurrency APIs, I tried a few times to implement different concurrency models and always failed due to shared requiring either too much casts or forces too many structures and methods to be shared. I vote for a 'lent' qualifier to be implemented in the compiler. Jeremie
Oct 12 2009
parent reply Jeremie Pelletier <jeremiep gmail.com> writes:
Leandro Lucarella wrote:
 Jeremie Pelletier, el 12 de octubre a las 22:45 me escribiste:
 I agree. Particularly about lent. Immutable and mutable weren't
 considered complete without const, so I'm surprised that local and
 shared are considered complete without lent. You can even
 implement it without escape analysis. Strangely, from what I
 remember of Bartosz's posts, it was unique that was the sticking
 point, though you can implement both unique and owned as library
 types (though they do become less efficient).

itself, I couldn't find something relevant on google.

The terms "unique", "shared" and "lent" used here generaly refers to the terms used by Bartoz Milewski in his blog[1]. I think he defines lent in this particular blog post[2], but maybe you might need to read a couple of posts more to understand everything. [1] http://bartoszmilewski.wordpress.com/ [2] http://bartoszmilewski.wordpress.com/2009/05/21/unique_ptr-how-unique-is-it/

I know of this article, he mentions the word but he doesn't say where it comes from. My native language is french (I'm from Quebec, Canada), I don't know every word of the english dictionary. That's why I asked, I'm just curious as to what it meant. I did some googling but I only landed on a religious page :/ I assume it means 'temporary shared' just like const is 'temporary immutable', but if i can make a link with the english language too its even better.
Oct 12 2009
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Jeremie Pelletier wrote:
 Leandro Lucarella wrote:
 Jeremie Pelletier, el 12 de octubre a las 22:45 me escribiste:
 I agree. Particularly about lent. Immutable and mutable weren't
 considered complete without const, so I'm surprised that local and
 shared are considered complete without lent. You can even
 implement it without escape analysis. Strangely, from what I
 remember of Bartosz's posts, it was unique that was the sticking
 point, though you can implement both unique and owned as library
 types (though they do become less efficient).

itself, I couldn't find something relevant on google.

The terms "unique", "shared" and "lent" used here generaly refers to the terms used by Bartoz Milewski in his blog[1]. I think he defines lent in this particular blog post[2], but maybe you might need to read a couple of posts more to understand everything. [1] http://bartoszmilewski.wordpress.com/ [2] http://bartoszmilewski.wordpress.com/2009/05/21/unique_ptr-how-unique-is-it/

I know of this article, he mentions the word but he doesn't say where it comes from. My native language is french (I'm from Quebec, Canada), I don't know every word of the english dictionary. That's why I asked, I'm just curious as to what it meant. I did some googling but I only landed on a religious page :/ I assume it means 'temporary shared' just like const is 'temporary immutable', but if i can make a link with the english language too its even better.

to lend loan lending are all of the same root. "lent" is the passive/simple past/past participle form of "to lend". I guess is the French word is one of "confer\'e" or "pr\^et\'e". Andrei P.S. Beer? Coffee? Marriage? I'll be in Quebec City on Dec 10th and 11th for a talk.
Oct 12 2009
next sibling parent reply Jeremie Pelletier <jeremiep gmail.com> writes:
Andrei Alexandrescu wrote:
 Jeremie Pelletier wrote:
 Leandro Lucarella wrote:
 Jeremie Pelletier, el 12 de octubre a las 22:45 me escribiste:
 I agree. Particularly about lent. Immutable and mutable weren't
 considered complete without const, so I'm surprised that local and
 shared are considered complete without lent. You can even
 implement it without escape analysis. Strangely, from what I
 remember of Bartosz's posts, it was unique that was the sticking
 point, though you can implement both unique and owned as library
 types (though they do become less efficient).

itself, I couldn't find something relevant on google.

The terms "unique", "shared" and "lent" used here generaly refers to the terms used by Bartoz Milewski in his blog[1]. I think he defines lent in this particular blog post[2], but maybe you might need to read a couple of posts more to understand everything. [1] http://bartoszmilewski.wordpress.com/ [2] http://bartoszmilewski.wordpress.com/2009/05/21/unique_ptr-how-unique-is-it/

I know of this article, he mentions the word but he doesn't say where it comes from. My native language is french (I'm from Quebec, Canada), I don't know every word of the english dictionary. That's why I asked, I'm just curious as to what it meant. I did some googling but I only landed on a religious page :/ I assume it means 'temporary shared' just like const is 'temporary immutable', but if i can make a link with the english language too its even better.

to lend loan lending are all of the same root. "lent" is the passive/simple past/past participle form of "to lend". I guess is the French word is one of "confer\'e" or "pr\^et\'e".

Thanks! I now make the link, may I then suggest the keyword 'borrow', seems to make more sense to me. Do you speak any french or did you just google that? :o)
 Andrei
 
 P.S. Beer? Coffee? Marriage? I'll be in Quebec City on Dec 10th and 11th 
 for a talk.

I will most definitely try to be there, I'd love a beer or a coffee, as for marriage.. you'd need to be female from birth, sorry :) Jeremie
Oct 12 2009
parent reply Jeremie Pelletier <jeremiep gmail.com> writes:
Robert Jacques wrote:
 On Tue, 13 Oct 2009 02:25:07 -0400, Jeremie Pelletier 
 <jeremiep gmail.com> wrote:
 Andrei Alexandrescu wrote:

  to lend
 loan
 lending
  are all of the same root. "lent" is the passive/simple past/past 
 participle form of "to lend".
  I guess is the French word is one of "confer\'e" or "pr\^et\'e".

Thanks! I now make the link, may I then suggest the keyword 'borrow', seems to make more sense to me.

Borrow is a verb, borrowed would be correct noun (i.e. past tense). A lent object and a borrowed object have practically the same meaning, but one word has half the number of characters.

I disagree that they have the same meaning, one side lends the object and the other borrows it :) But I agree that it makes more sense after reading what you said, maybe I just don't like the sound of past tense verbs in programming keywords, are there any other such keywords in D? Isn't there a qualifier name that would means lent or borrowed without being past-tense, without being a verb implying it also is a function (such as assert). Jeremie
Oct 12 2009
parent Jeremie Pelletier <jeremiep gmail.com> writes:
Robert Jacques wrote:
 On Tue, 13 Oct 2009 02:50:41 -0400, Jeremie Pelletier 
 <jeremiep gmail.com> wrote:
 
 Robert Jacques wrote:
 On Tue, 13 Oct 2009 02:25:07 -0400, Jeremie Pelletier 
 <jeremiep gmail.com> wrote:
 Andrei Alexandrescu wrote:

  to lend
 loan
 lending
  are all of the same root. "lent" is the passive/simple past/past 
 participle form of "to lend".
  I guess is the French word is one of "confer\'e" or "pr\^et\'e".

Thanks! I now make the link, may I then suggest the keyword 'borrow', seems to make more sense to me.

A lent object and a borrowed object have practically the same meaning, but one word has half the number of characters.

I disagree that they have the same meaning, one side lends the object and the other borrows it :) But I agree that it makes more sense after reading what you said, maybe I just don't like the sound of past tense verbs in programming keywords, are there any other such keywords in D?

shared?

I stand corrected :x
Oct 13 2009
prev sibling parent Michel Fortin <michel.fortin michelf.com> writes:
On 2009-10-13 02:16:30 -0400, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:

 P.S. Beer? Coffee? Marriage? I'll be in Quebec City on Dec 10th and 
 11th for a talk.

So you'll be in my city too. :-) How many of us are in Quebec City? -- Michel Fortin michel.fortin michelf.com http://michelf.com/
Oct 13 2009
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 13 Oct 2009 01:59:57 -0400, Jeremie Pelletier <jeremiep gmail.com>  
wrote:

 Leandro Lucarella wrote:
 Jeremie Pelletier, el 12 de octubre a las 22:45 me escribiste:
 I agree. Particularly about lent. Immutable and mutable weren't
 considered complete without const, so I'm surprised that local and
 shared are considered complete without lent. You can even
 implement it without escape analysis. Strangely, from what I
 remember of Bartosz's posts, it was unique that was the sticking
 point, though you can implement both unique and owned as library
 types (though they do become less efficient).

itself, I couldn't find something relevant on google.

the terms used by Bartoz Milewski in his blog[1]. I think he defines lent in this particular blog post[2], but maybe you might need to read a couple of posts more to understand everything. [1] http://bartoszmilewski.wordpress.com/ [2] http://bartoszmilewski.wordpress.com/2009/05/21/unique_ptr-how-unique-is-it/

I know of this article, he mentions the word but he doesn't say where it comes from. My native language is french (I'm from Quebec, Canada), I don't know every word of the english dictionary. That's why I asked, I'm just curious as to what it meant. I did some googling but I only landed on a religious page :/ I assume it means 'temporary shared' just like const is 'temporary immutable', but if i can make a link with the english language too its even better.

Oh. Yes Lent is a religious holiday, but it's also simple past tense and past participle of lend. ( From http://en.wiktionary.org/wiki/lent ) Apparently it's based off of lentus from French.
Oct 12 2009
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 13 Oct 2009 02:25:07 -0400, Jeremie Pelletier <jeremiep gmail.com>  
wrote:
 Andrei Alexandrescu wrote:

  to lend
 loan
 lending
  are all of the same root. "lent" is the passive/simple past/past  
 participle form of "to lend".
  I guess is the French word is one of "confer\'e" or "pr\^et\'e".

Thanks! I now make the link, may I then suggest the keyword 'borrow', seems to make more sense to me.

Borrow is a verb, borrowed would be correct noun (i.e. past tense). A lent object and a borrowed object have practically the same meaning, but one word has half the number of characters.
Oct 12 2009
prev sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 13 Oct 2009 02:50:41 -0400, Jeremie Pelletier <jeremiep gmail.com>  
wrote:

 Robert Jacques wrote:
 On Tue, 13 Oct 2009 02:25:07 -0400, Jeremie Pelletier  
 <jeremiep gmail.com> wrote:
 Andrei Alexandrescu wrote:

  to lend
 loan
 lending
  are all of the same root. "lent" is the passive/simple past/past  
 participle form of "to lend".
  I guess is the French word is one of "confer\'e" or "pr\^et\'e".

Thanks! I now make the link, may I then suggest the keyword 'borrow', seems to make more sense to me.

lent object and a borrowed object have practically the same meaning, but one word has half the number of characters.

I disagree that they have the same meaning, one side lends the object and the other borrows it :) But I agree that it makes more sense after reading what you said, maybe I just don't like the sound of past tense verbs in programming keywords, are there any other such keywords in D?

shared?
 Isn't there a qualifier name that would means lent or borrowed without  
 being past-tense, without being a verb implying it also is a function  
 (such as assert).

 Jeremie

Oct 12 2009
prev sibling next sibling parent Leandro Lucarella <llucax gmail.com> writes:
Jeremie Pelletier, el 12 de octubre a las 22:45 me escribiste:
I agree. Particularly about lent. Immutable and mutable weren't
considered complete without const, so I'm surprised that local and
shared are considered complete without lent. You can even
implement it without escape analysis. Strangely, from what I
remember of Bartosz's posts, it was unique that was the sticking
point, though you can implement both unique and owned as library
types (though they do become less efficient).

I may sound ignorant, but that does lent means? I mean the word itself, I couldn't find something relevant on google.

The terms "unique", "shared" and "lent" used here generaly refers to the terms used by Bartoz Milewski in his blog[1]. I think he defines lent in this particular blog post[2], but maybe you might need to read a couple of posts more to understand everything. [1] http://bartoszmilewski.wordpress.com/ [2] http://bartoszmilewski.wordpress.com/2009/05/21/unique_ptr-how-unique-is-it/ -- Leandro Lucarella (AKA luca) http://llucax.com.ar/ ---------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ---------------------------------------------------------------------- Sometimes you got to suffer a little in your youth to motivate you to succeed later in life. Do you think if Bill Gates got laid in high school, do you think there'd be a Microsoft? Of course not. You gotta spend a lot of time stuffin your own locker with your underwear wedged up your arse before you think "I'm gona take over the world with computers! You'll see I'll show them."
Oct 12 2009
prev sibling parent Leandro Lucarella <llucax gmail.com> writes:
Jeremie Pelletier, el 13 de octubre a las 01:59 me escribiste:
 Leandro Lucarella wrote:
Jeremie Pelletier, el 12 de octubre a las 22:45 me escribiste:
I agree. Particularly about lent. Immutable and mutable weren't
considered complete without const, so I'm surprised that local and
shared are considered complete without lent. You can even
implement it without escape analysis. Strangely, from what I
remember of Bartosz's posts, it was unique that was the sticking
point, though you can implement both unique and owned as library
types (though they do become less efficient).

itself, I couldn't find something relevant on google.

The terms "unique", "shared" and "lent" used here generaly refers to the terms used by Bartoz Milewski in his blog[1]. I think he defines lent in this particular blog post[2], but maybe you might need to read a couple of posts more to understand everything. [1] http://bartoszmilewski.wordpress.com/ [2] http://bartoszmilewski.wordpress.com/2009/05/21/unique_ptr-how-unique-is-it/

I know of this article, he mentions the word but he doesn't say where it comes from. My native language is french (I'm from Quebec, Canada), I don't know every word of the english dictionary. That's why I asked, I'm just curious as to what it meant.

Woops! Sorry, I understood exactly the opposite from you mail =/ -- Leandro Lucarella (AKA luca) http://llucax.com.ar/ ---------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ---------------------------------------------------------------------- Sus discipulos se miraron sin entended hasta que uno preguntose: Peperino, soy Daniel Q. de Olivos tengo 54 años y aún soy virgen. A lo que Peperino respondiole: Si sos ganso, ganso ser. Y lo frotó, y lo curó y lo sanó. A lo que todos dijeron: ­¡¡¡Peperino se la come, Peperino se la come!!! -- Peperino Pómoro
Oct 13 2009