www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Re: object splitting in multiple threads

reply yes <yes no.com> writes:
 
 What does this code do? It looks like a ThreadPool use case but I might be
wrong.
 

I tried to recursively make an object distribute its calculations. The calculations should take at least a minute.
Jan 09 2009
parent reply "Denis Koroskin" <2korden gmail.com> writes:
On Sat, 10 Jan 2009 07:32:43 +0300, yes <yes no.com> wrote:

 What does this code do? It looks like a ThreadPool use case but I might  
 be wrong.

I tried to recursively make an object distribute its calculations. The calculations should take at least a minute.

Hm... I guess I understand now, you need a task parallelization, right? In order to do this, you usually separate the task into several independent sub-tasks, put into a queue and wait until it is finished. For example, we have a list of files to download from remote server. For each file, we create a new connection and retrieve it independently (and, possibly, out of order): import tango.core.ThreadPool; void execute() { auto pool = new ThreadPool!(string)(10); // create a thread pool with 10 threads string[] fileList = loadFileList(); foreach (string filePath; fileList) { pool.append(&download, filePath); // append the task } pool.finish(); // wait until all the tasks complete } void download(string fileName) { // ... } In this example, I create a lot of threads (more than hardware runs concurrently) because the CPU is not a bottleneck. You may experience different results in different CPU workaload, though. Hope that helps.
Jan 09 2009
next sibling parent reply yes <yes no.com> writes:
Does Phobos also do threadpools?

Somehow I liked the idea of an Object like agent smith, just duplicate yourself
when the task is too big. ( ^_^ )

 
 
 Hm... I guess I understand now, you need a task parallelization, right?
 
 In order to do this, you usually separate the task into several independent
sub-tasks, put into a queue and wait until it is finished.
 
 For example, we have a list of files to download from remote server. For each
file, we create a new connection and retrieve it independently (and, possibly,
out of order):
 
 import tango.core.ThreadPool;
 
 void execute()
 {
     auto pool = new ThreadPool!(string)(10); // create a thread pool with 10
threads
     string[] fileList = loadFileList();
     foreach (string filePath; fileList) {
         pool.append(&download, filePath);   // append the task
     }
 
     pool.finish(); // wait until all the tasks complete
 }
 
 void download(string fileName)
 {
     // ...
 }
 
 In this example, I create a lot of threads (more than hardware runs
concurrently) because the CPU is not a bottleneck.
 You may experience different results in different CPU workaload, though.
 
 Hope that helps.
 

Jan 09 2009
parent downs <default_357-line yahoo.de> writes:
Jarrett Billingsley wrote:
 On Sat, Jan 10, 2009 at 12:30 AM, Jarrett Billingsley
 <jarrett.billingsley gmail.com> wrote:
 On Sat, Jan 10, 2009 at 12:18 AM, yes <yes no.com> wrote:
 Does Phobos also do threadpools?

efficiently. Downs has implemented such a thing in scrapple.tools, but it requires a patch to Phobos to make it run at a reasonable speed.

Ffffffff.. ignore this, I was thinking about fiber pools. I don't think there's any technical reason why a ThreadPool couldn't be written for Phobos, but there isn't one provided by default.

Scrapple.Tools has a Threadpool too :)
Jan 10 2009
prev sibling next sibling parent "Jarrett Billingsley" <jarrett.billingsley gmail.com> writes:
On Sat, Jan 10, 2009 at 12:30 AM, Jarrett Billingsley
<jarrett.billingsley gmail.com> wrote:
 On Sat, Jan 10, 2009 at 12:18 AM, yes <yes no.com> wrote:
 Does Phobos also do threadpools?

From what I understand, not without modification. At least, not efficiently. Downs has implemented such a thing in scrapple.tools, but it requires a patch to Phobos to make it run at a reasonable speed.

Ffffffff.. ignore this, I was thinking about fiber pools. I don't think there's any technical reason why a ThreadPool couldn't be written for Phobos, but there isn't one provided by default.
Jan 09 2009
prev sibling next sibling parent "Jarrett Billingsley" <jarrett.billingsley gmail.com> writes:
On Sat, Jan 10, 2009 at 12:18 AM, yes <yes no.com> wrote:
 Does Phobos also do threadpools?

From what I understand, not without modification. At least, not efficiently. Downs has implemented such a thing in scrapple.tools, but it requires a patch to Phobos to make it run at a reasonable speed.
 Somehow I liked the idea of an Object like agent smith, just duplicate
yourself when the task is too big. ( ^_^ )

Hehe.
Jan 09 2009
prev sibling parent "Jarrett Billingsley" <jarrett.billingsley gmail.com> writes:
On Sat, Jan 10, 2009 at 6:39 AM, downs <default_357-line yahoo.de> wrote:
 Jarrett Billingsley wrote:
 On Sat, Jan 10, 2009 at 12:30 AM, Jarrett Billingsley
 <jarrett.billingsley gmail.com> wrote:
 On Sat, Jan 10, 2009 at 12:18 AM, yes <yes no.com> wrote:
 Does Phobos also do threadpools?

efficiently. Downs has implemented such a thing in scrapple.tools, but it requires a patch to Phobos to make it run at a reasonable speed.

Ffffffff.. ignore this, I was thinking about fiber pools. I don't think there's any technical reason why a ThreadPool couldn't be written for Phobos, but there isn't one provided by default.

Scrapple.Tools has a Threadpool too :)

NO! *SLAM*
Jan 10 2009