www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Maximum number of threads

reply Alex <sascha.orlov gmail.com> writes:
This should be a not so long question to answer, I hope.

I took an example from the "Programming in D" book, chapter 
"Message Passing Concurrency", around page 550. The question of 
interest was, how many threads I can have spawned at the same 
time.
So I made an array of robot objects from the example and found 
out: the maximum number is different on different PCs I have.
For example:
- on my mac I can have 2048 threads spawned at the same time
- on my linux machine the maximum number is 32192.
The numbers are quite fixed, however there were some small 
fluctuations on the linux machine.

The questions still remains: how do I know, what maximum number 
of threads I can have? Is it possible to calculate this value 
during runtime of my program? The two machines I have available 
for testing are very different, what is the main parameter which 
controls the number of possible threads? (CPU maybe? Number of 
cores? RAM is not I think at the moment...)

Thanks in advance
Alex
Sep 24 2015
next sibling parent Marc =?UTF-8?B?U2Now7x0eg==?= <schuetzm gmx.net> writes:
On Thursday, 24 September 2015 at 08:55:25 UTC, Alex wrote:

 This should be a not so long question to answer, I hope.

 I took an example from the "Programming in D" book, chapter 
 "Message Passing Concurrency", around page 550. The question of 
 interest was, how many threads I can have spawned at the same 
 time.
 So I made an array of robot objects from the example and found 
 out: the maximum number is different on different PCs I have.
 For example:
 - on my mac I can have 2048 threads spawned at the same time
 - on my linux machine the maximum number is 32192.
 The numbers are quite fixed, however there were some small 
 fluctuations on the linux machine.

 The questions still remains: how do I know, what maximum number 
 of threads I can have? Is it possible to calculate this value 
 during runtime of my program? The two machines I have available 
 for testing are very different, what is the main parameter 
 which controls the number of possible threads? (CPU maybe? 
 Number of cores? RAM is not I think at the moment...)

 Thanks in advance
 Alex


 From the numbers (2048 = 2^11, 32192 = ca 2^15), it's clear that 
you're not running out of memory, but that these are artificial 
limits imposed by the OS.

This stackoverflow answer describes the situation on Linux:
http://unix.stackexchange.com/a/47600/34768
Sep 24 2015
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d-learn writes:
On Thursday, September 24, 2015 08:55:22 Alex via Digitalmars-d-learn wrote:

 This should be a not so long question to answer, I hope.

 I took an example from the "Programming in D" book, chapter
 "Message Passing Concurrency", around page 550. The question of
 interest was, how many threads I can have spawned at the same
 time.
 So I made an array of robot objects from the example and found
 out: the maximum number is different on different PCs I have.
 For example:
 - on my mac I can have 2048 threads spawned at the same time
 - on my linux machine the maximum number is 32192.
 The numbers are quite fixed, however there were some small
 fluctuations on the linux machine.

 The questions still remains: how do I know, what maximum number
 of threads I can have? Is it possible to calculate this value
 during runtime of my program? The two machines I have available
 for testing are very different, what is the main parameter which
 controls the number of possible threads? (CPU maybe? Number of
 cores? RAM is not I think at the moment...)


It's entirely system specific. Not only does every OS handle it differently,
but it con depend on how your machine is configured. In general though, I
believe that it's a hard limit in the OS and has nothing to do with the
number of cores or amount of RAM in your machine (though even if you haven't
hit the limit, if the OS doesn't have enough resources to create another
thread, then it will fail).

So, if you want to know for a particular OS, you're probably going to have
to google for it. In the case of linux, a quick search seems to indicate
that getrlimit will do the trick:

http://linux.die.net/man/2/getrlimit

But you can find as much pretty quickly googling yourself. Maybe someone
else can tell you what to use on other OSes off the top of their head
though.

Regardless, D doesn't provid any special way to do this, so to figure it
out, you'd be calling whatever the C functions are that will tell you.

- Jonathan M Davis
Sep 24 2015
prev sibling parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Thursday, 24 September 2015 at 08:55:25 UTC, Alex wrote:

 - on my mac I can have 2048 threads spawned at the same time
 - on my linux machine the maximum number is 32192.
 The numbers are quite fixed, however there were some small 
 fluctuations on the linux machine.


I only know of `cat /proc/sys/kernel/threads-max` on linux, which 
will give you the max number of threads system-wide.
Sep 24 2015
parent reply Temtaime <temtaime gmail.com> writes:
Offtop: i think if number of threads > number of real cores, than 
there's something wrong with your design. Maybe fibers suit 
better ?
Sep 24 2015
next sibling parent Jonathan M Davis via Digitalmars-d-learn writes:
On Thursday, September 24, 2015 12:38:39 Temtaime via Digitalmars-d-learn wrote:

 Offtop: i think if number of threads > number of real cores, than
 there's something wrong with your design. Maybe fibers suit
 better ?


That depends on what the threads are doing. If they're all CPU-intensive,
then yeah, having more than about the number of real cores isn't going to
work very well, but in many programs, a lot of the threads aren't doing much
(e.g. waiting for something to happen and sitting idly in the meantime). If
anything, I'd guess that it's far more common to have a lot more threads
than cores if a program isn't single-threaded. I'd say that whether the
number of cores matters particularly depends heavily on what your program is
doing. But there are certainly cases where using fibers instead of threads
makes a lot of sense.

- Jonathan M Davis
Sep 24 2015
prev sibling parent Alex <sascha.orlov gmail.com> writes:
On Thursday, 24 September 2015 at 12:38:41 UTC, Temtaime wrote:

 Offtop: i think if number of threads > number of real cores, 
 than there's something wrong with your design. Maybe fibers 
 suit better ?


Well... you got my idea :) So it is not so far offtop, as you 
think )))
Fibers DO suit better as an idea of light workers. But the 
problem I have to manage is, that my program has to incorporate 
unpredictability in the work cycle of my workers. It is one of 
the main idea of the process itself. So the fact that a fiber 
yields at fixed points of his work is just the opposite from what 
I want.

But the portability of code is an issue too... and my tries show, 
that I can't rely on this with threads...

Having said that, I got an idea today, how I could try to 
incorporate fibers for my needs. If I manage to hide the yields 
calls of my workers in some random environment, I could express, 
that a worker can but must not give up his control at a specific 
point of his work. I have to try this idea next days.
Using fibers instead of threads would save me from a lot of 
troubles anyway... ;)

Thanks to everybody for hints about my question!
Sep 24 2015