www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Channels for tasks?

reply "Matt" <MATTCA sky.com> writes:
Hi guys,
I know this will have probably been answered before, but does D 
have a concept similar to Go's channels when it comes to tasks?

I believe (according to a few forum posts hinting at it) that 
such a thing exists for threads, but not for the lightweight 
tasks provided by std.parallelism. Are there any plans to add 
such a feature to the library?

If not, how would I go about implementing them in D? I imagine I 
would use some form of blocking list, but  I've not done much 
research on how they are done in Go/Rust/etc.

Thanks,
Matt.

P.S. Sorry if this is in the wrong sub-forum, I'll be happy for 
mods to move it if necessary.
Jul 18 2013
next sibling parent "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Thursday, 18 July 2013 at 15:29:00 UTC, Matt wrote:
 Hi guys,
 I know this will have probably been answered before, but does D 
 have a concept similar to Go's channels when it comes to tasks?

I don't think this is really an answer, but awhile back I was looking into the difference of D and Go for parallel/concurrency and created these two examples[1][2] from the blog http://www.mprescient.com/journal/2011/1/9/concurrency-in-go-a-call-center-tutorial.html 1. https://gist.github.com/JesseKPhillips/773979 2. https://gist.github.com/JesseKPhillips/774983 The first is a translation to message passing, the second is using std.parallelism I think there is room for higher level APIs, but I don't think we will see anything that resembles Goroutines being added.
 P.S. Sorry if this is in the wrong sub-forum, I'll be happy for 
 mods to move it if necessary.

Should go to D.learn, but not a huge deal.
Jul 18 2013
prev sibling next sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
On Jul 18, 2013, at 8:28 AM, Matt <MATTCA sky.com> wrote:

 Hi guys,
 I know this will have probably been answered before, but does D have a =

=20
 I believe (according to a few forum posts hinting at it) that such a =

std.parallelism. Are there any plans to add such a feature to the = library? Not as such. I'd like to make Fibers each have their own message queue = in std.concurrency, but that means making TLS work at the fiber level, = which is tricky. I think there is value in the CSP model (ie. channels), but haven't done = anything about it in terms of library work.=
Jul 18 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-07-18 21:08, Sean Kelly wrote:

 We already do in-library TLS for OSX and so it shouldn't be terribly difficult
to use this logic for fibers, but things get tricky once you consider dynamic
libraries.

To be able to support dynamic libraries and TLS on Mac OS X we will most likely need to switch to the native implementation of TLS, supported in Mac OS X 10.7 and later. Or we would have to copy/mimic the TLS related code from the dynamic loader into druntime. -- /Jacob Carlborg
Jul 18 2013
prev sibling next sibling parent "Matt" <MATTCA sky.com> writes:
On Thursday, 18 July 2013 at 16:41:34 UTC, Jesse Phillips wrote:
 I don't think this is really an answer, but awhile back I was 
 looking into the difference of D and Go for 
 parallel/concurrency and created these two examples[1][2] from 
 the blog

 http://www.mprescient.com/journal/2011/1/9/concurrency-in-go-a-call-center-tutorial.html

 1. https://gist.github.com/JesseKPhillips/773979
 2. https://gist.github.com/JesseKPhillips/774983

 The first is a translation to message passing, the second is 
 using std.parallelism

 I think there is room for higher level APIs, but I don't think 
 we will see anything that resembles Goroutines being added.

Thanks for the examples, I'll have to take a look at them later! I guess the different backgrounds is the reason for a lack of D 'goroutines', but parallelism seemed a good alternative. Is the performance of parallelism and it's taskpools similar to goroutines? It seems they both do the same thing (adding the tasks to a queue, allocating them to a group of threads, etc), but does the level of implementation (library vs language) mean a speed difference? Furthermore, is there a limit to the number of tasks that can be spawned and performed in parallel? I know Rust allows for hundreds, even thousands, of tasks to be executed "concurrently." Thanks for the replies, Matt.
Jul 18 2013
prev sibling next sibling parent "Matt" <MATTCA sky.com> writes:
On Thursday, 18 July 2013 at 17:55:36 UTC, Sean Kelly wrote:
 Not as such.  I'd like to make Fibers each have their own 
 message queue in std.concurrency, but that means making TLS 
 work at the fiber level, which is tricky.

 I think there is value in the CSP model (ie. channels), but 
 haven't done anything about it in terms of library work.

From what I've read about Fibers, it seems they are more similar to what I'm looking for (a post on stackoverflow implied they are similar to Erlang's implementation of lightweight tasks). Does this mean to say they are similar to the tasks provided by std.parallelism? Thanks for the reply, Matt.
Jul 18 2013
prev sibling next sibling parent "Mr. Anonymous" <mailnew4ster gmail.com> writes:
On Thursday, 18 July 2013 at 18:38:57 UTC, Matt wrote:
 On Thursday, 18 July 2013 at 17:55:36 UTC, Sean Kelly wrote:
 Not as such.  I'd like to make Fibers each have their own 
 message queue in std.concurrency, but that means making TLS 
 work at the fiber level, which is tricky.

 I think there is value in the CSP model (ie. channels), but 
 haven't done anything about it in terms of library work.

From what I've read about Fibers, it seems they are more similar to what I'm looking for (a post on stackoverflow implied they are similar to Erlang's implementation of lightweight tasks). Does this mean to say they are similar to the tasks provided by std.parallelism? Thanks for the reply, Matt.

vibe.d is based on fibers, you might want to check it out. http://vibed.org/features#fibers
Jul 18 2013
prev sibling next sibling parent Sean Kelly <sean invisibleduck.org> writes:
On Jul 18, 2013, at 11:38 AM, "Matt" <MATTCA sky.com> wrote:

 On Thursday, 18 July 2013 at 17:55:36 UTC, Sean Kelly wrote:
 Not as such.  I'd like to make Fibers each have their own message =


level, which is tricky.
=20
 I think there is value in the CSP model (ie. channels), but haven't =


=20
 =46rom what I've read about Fibers, it seems they are more similar to =

to Erlang's implementation of lightweight tasks). Does this mean to say = they are similar to the tasks provided by std.parallelism? Functionally, fibers are coroutines. They have their own stack and = execute within the context of the calling thread. Most languages that = scale to thousands or millions of concurrent threads/processes use = fibers (often called "green threads" in this context) to pull it off. = The obstacle for D is that global data is thread-local by default, so if = we were to use fibers in the back-end to allow the creation of a large = number of "threads", fibers would basically need their own TLS to avoid = breaking code. We already do in-library TLS for OSX and so it shouldn't = be terribly difficult to use this logic for fibers, but things get = tricky once you consider dynamic libraries. It really should happen at = some point though, for std.concurrency to operate at its full potential.=
Jul 18 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 18 July 2013 at 17:55:36 UTC, Sean Kelly wrote:
 On Jul 18, 2013, at 8:28 AM, Matt <MATTCA sky.com> wrote:

 Hi guys,
 I know this will have probably been answered before, but does 
 D have a concept similar to Go's channels when it comes to 
 tasks?
 
 I believe (according to a few forum posts hinting at it) that 
 such a thing exists for threads, but not for the lightweight 
 tasks provided by std.parallelism. Are there any plans to add 
 such a feature to the library?

Not as such. I'd like to make Fibers each have their own message queue in std.concurrency, but that means making TLS work at the fiber level, which is tricky.

Yes please yes. Did I said yes yet ?
 I think there is value in the CSP model (ie. channels), but 
 haven't done anything about it in terms of library work.

Jul 19 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 18 July 2013 at 19:08:25 UTC, Sean Kelly wrote:
 Functionally, fibers are coroutines.  They have their own stack 
 and execute within the context of the calling thread.  Most 
 languages that scale to thousands or millions of concurrent 
 threads/processes use fibers (often called "green threads" in 
 this context) to pull it off.  The obstacle for D is that 
 global data is thread-local by default, so if we were to use 
 fibers in the back-end to allow the creation of a large number 
 of "threads", fibers would basically need their own TLS to 
 avoid breaking code.  We already do in-library TLS for OSX and 
 so it shouldn't be terribly difficult to use this logic for 
 fibers, but things get tricky once you consider dynamic 
 libraries.  It really should happen at some point though, for 
 std.concurrency to operate at its full potential.

My take is that we should keep fibers as they are. But change Thread (as the class in the runtime) to be Fibers with FLS and then let the runtime operate system thread and schedule Thread on them. The read system thread can still be provided in some dark corner of the standard lib or runtime.
Jul 19 2013
prev sibling next sibling parent Sean Kelly <sean invisibleduck.org> writes:
On Jul 19, 2013, at 12:48 AM, deadalnix <deadalnix gmail.com> wrote:

 On Thursday, 18 July 2013 at 19:08:25 UTC, Sean Kelly wrote:
 Functionally, fibers are coroutines.  They have their own stack and =


scale to thousands or millions of concurrent threads/processes use = fibers (often called "green threads" in this context) to pull it off. = The obstacle for D is that global data is thread-local by default, so if = we were to use fibers in the back-end to allow the creation of a large = number of "threads", fibers would basically need their own TLS to avoid = breaking code. We already do in-library TLS for OSX and so it shouldn't = be terribly difficult to use this logic for fibers, but things get = tricky once you consider dynamic libraries. It really should happen at = some point though, for std.concurrency to operate at its full potential.
=20
 My take is that we should keep fibers as they are. But change Thread =

runtime operate system thread and schedule Thread on them. I think this would most likely happen within std.concurrency, with the = context switch occurring on send/receive.=
Jul 22 2013
prev sibling next sibling parent Sean Kelly <sean invisibleduck.org> writes:
On Jul 18, 2013, at 11:35 PM, Jacob Carlborg <doob me.com> wrote:

 On 2013-07-18 21:08, Sean Kelly wrote:
=20
 We already do in-library TLS for OSX and so it shouldn't be terribly =


consider dynamic libraries.
=20
 To be able to support dynamic libraries and TLS on Mac OS X we will =

supported in Mac OS X 10.7 and later. Or we would have to copy/mimic the = TLS related code from the dynamic loader into druntime. And the same for other OSes as well, if we started doing manual TLS for = fibers. Super not fun. For reference, here's how TLS works on ELF: http://www.akkadia.org/drepper/tls.pdf=
Jul 22 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Monday, 22 July 2013 at 18:24:53 UTC, Sean Kelly wrote:
 I think this would most likely happen within std.concurrency, 
 with the context switch occurring on send/receive.

If so, other libraries won't be able to yield (for instance libraries performing IO).
Jul 22 2013
prev sibling next sibling parent "Dejan Lekic" <dejan.lekic gmail.com> writes:
On Friday, 19 July 2013 at 07:42:26 UTC, deadalnix wrote:
 On Thursday, 18 July 2013 at 17:55:36 UTC, Sean Kelly wrote:
 On Jul 18, 2013, at 8:28 AM, Matt <MATTCA sky.com> wrote:

 Hi guys,
 I know this will have probably been answered before, but does 
 D have a concept similar to Go's channels when it comes to 
 tasks?
 
 I believe (according to a few forum posts hinting at it) that 
 such a thing exists for threads, but not for the lightweight 
 tasks provided by std.parallelism. Are there any plans to add 
 such a feature to the library?

Not as such. I'd like to make Fibers each have their own message queue in std.concurrency, but that means making TLS work at the fiber level, which is tricky.

Yes please yes. Did I said yes yet ?
 I think there is value in the CSP model (ie. channels), but 
 haven't done anything about it in terms of library work.


+1
Jul 23 2013
prev sibling parent Sean Kelly <sean invisibleduck.org> writes:
On Jul 22, 2013, at 11:20 PM, deadalnix <deadalnix gmail.com> wrote:

 On Monday, 22 July 2013 at 18:24:53 UTC, Sean Kelly wrote:
 I think this would most likely happen within std.concurrency, with =


=20
 If so, other libraries won't be able to yield (for instance libraries =

Yes, there would have to be a yield method available for library = writers.
Jul 23 2013