www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - 10's of threads + network == segfault (sample code)

reply prefetch <prefetch_member pathlink.com> writes:
not sure if this is the right place to post this, let me know if i should be
posting this somewhere else.

please try running the code below and see if you get segfaults every 5th or 6th
try.  not exactly sure what is going on or what to blame.

i tried running in gdb, but i can't reproduce it there.

has anyone used attempted to build very large systems (hundreds of concurrent
threads) using D?

let me know if you are able to reproduce this segfault (it should only take
about 10 times or so with a few dozen threads as the arg.)  i'd really
appreciate it.

----8<-----

import std.string, std.conv, std.stream;
import std.thread, std.socket, std.socketstream;

int netthread(void*nothing) {
Socket sock = new TcpSocket(new InternetAddress(server, port));
Stream ss = new SocketStream(sock);
/* do something with sockets */
return 0;
}

uint port = 8080;
char[] server = "localhost";

/* run this with 10 or so threads, and it will seg fault after a few tries. */
/* Linux 2.6.8-2-686-smp i686 GNU/Linux */
/* dmd v0.129 */

int main(char[][] args)
{
if(args.length < 2)
{
printf("Usage:\n   netthreadcrash numthreads\n");
return 0;
}
uint numthreads = std.conv.toUint(args[1]);

int i = 0;
while (i++ < numthreads) {
Thread mythread = new Thread(&netthread, null);
mythread.start();
}
return 0;
}
Aug 17 2005
parent reply "Ben Hinkle" <bhinkle mathworks.com> writes:
"prefetch" <prefetch_member pathlink.com> wrote in message 
news:ddvq51$2q0$1 digitaldaemon.com...
 not sure if this is the right place to post this, let me know if i should 
 be
 posting this somewhere else.

 please try running the code below and see if you get segfaults every 5th 
 or 6th
 try.  not exactly sure what is going on or what to blame.

 i tried running in gdb, but i can't reproduce it there.

 has anyone used attempted to build very large systems (hundreds of 
 concurrent
 threads) using D?

 let me know if you are able to reproduce this segfault (it should only 
 take
 about 10 times or so with a few dozen threads as the arg.)  i'd really
 appreciate it.

 ----8<-----

 import std.string, std.conv, std.stream;
 import std.thread, std.socket, std.socketstream;

 int netthread(void*nothing) {
 Socket sock = new TcpSocket(new InternetAddress(server, port));
 Stream ss = new SocketStream(sock);
 /* do something with sockets */
 return 0;
 }

 uint port = 8080;
 char[] server = "localhost";

 /* run this with 10 or so threads, and it will seg fault after a few 
 tries. */
 /* Linux 2.6.8-2-686-smp i686 GNU/Linux */
 /* dmd v0.129 */

 int main(char[][] args)
 {
 if(args.length < 2)
 {
 printf("Usage:\n   netthreadcrash numthreads\n");
 return 0;
 }
 uint numthreads = std.conv.toUint(args[1]);

 int i = 0;
 while (i++ < numthreads) {
 Thread mythread = new Thread(&netthread, null);
 mythread.start();
 }
 return 0;
 }

D doesn't wait for child threads to finish before trying to do a gc and exit. This causes nasty seg-v's on exit. Get around the problem by making sure all child threads are done before returning from main(). See for example http://www.digitalmars.com/d/archives/digitalmars/D/13528.html and http://www.digitalmars.com/d/archives/digitalmars/D/6419.html and probably more.
Aug 17 2005
parent reply prefetch <prefetch_member pathlink.com> writes:
thanks ben.  sounds like D is broken in this regard.  exiting main w/o cleaning
up threads should not result in a segfault.  just to make supre duper sure, i
wrote a quick test in C using pthreads, and it has no problems.

so, is there a D bug tracker?
how do i submit a bug?

i know there is a D bug newsgroup, but i'm assuming that's a discussion forum
and not an actual bug tracking system (i hope..)

In article <ddvv95$77p$1 digitaldaemon.com>, Ben Hinkle says...

D doesn't wait for child threads to finish before trying to do a gc and 
exit. This causes nasty seg-v's on exit. Get around the problem by making 
sure all child threads are done before returning from main().
See for example 
http://www.digitalmars.com/d/archives/digitalmars/D/13528.html and 
http://www.digitalmars.com/d/archives/digitalmars/D/6419.html and probably 
more. 

Aug 17 2005
next sibling parent reply Derek Parnell <derek psych.ward> writes:
On Wed, 17 Aug 2005 21:10:46 +0000 (UTC), prefetch wrote:


 so, is there a D bug tracker?
 how do i submit a bug?
 
 i know there is a D bug newsgroup, but i'm assuming that's a discussion forum
 and not an actual bug tracking system (i hope..)

Sorry to be the bearer of bad news, but digitalmars.D.bugs *is* the bug tracking 'system'. Fortunately, Thomas Kühne is helping us with the 'tracking' part. -- Derek Parnell Melbourne, Australia 18/08/2005 7:16:40 AM
Aug 17 2005
parent reply prefetch <prefetch_member pathlink.com> writes:
In article <1bl4gtbuuy6ek.12qm3vcv0nnxa$.dlg 40tude.net>, Derek Parnell says...

Sorry to be the bearer of bad news, but digitalmars.D.bugs *is* the bug
tracking 'system'. Fortunately, Thomas Kühne is helping us with the
'tracking' part.

holy crap batman, who is driving this boat?? ;-) how on earth can you expect to develop a language and runtime library without a bug tracking system? i mean, this is just plain stupid. we all know, there are a whole bunch of free, web based bug trackers. don't get me wrong, i've really enjoyed learning about D, and i think it's a stellar language, but COME ON PEOPLE - let's get it together here. somewhere on the list of "how to start a super successful grassroots technology" is 'setup a web based bug tracking system so the community can help identify and fix bugs for the primary developer(s).' are my expectations just way to high or something? shouldn't this project reflect the quality of the language concept? i mean, it's a great concept - but how about we help it not fade into obscurity before it even gets going by building tools (docs, bug tracker, etc.) that can allow the community to grow?
Aug 17 2005
next sibling parent reply Manfred Nowak <svv1999 hotmail.com> writes:
prefetch <prefetch_member pathlink.com> wrote:

[...]
 holy crap batman, who is driving this boat?? ;-)

This superfast offshore-racer is punched through the waves solely by Walter Bright. Because of the mere mass of tiny fish that is spread all over the water, wanting the boat to race even faster and Walters limited capabilities only fish that is able to mimic a big whale pointing to fast routes through dangerous cliffs and shallows will be recognized. Furthermore everything ever written from that tiny fish becomes immediately a private property of Walters DigitalMars, except expressively stated otherwise. Therefore you just presnted another famous creature to become a private property :-) -manfred
Aug 17 2005
parent reply prefetch <prefetch_member pathlink.com> writes:
hm.  

so let me translate: 

1. "this is a one man show."
2. "if you want the one man to do something, you better figure out how to get
his attention."
3. "anything you do to help becomes private property of the one man show."

is this correct?

i snooped around and couldn't find much licensing info - except the compiler
comes with a license.txt file that implies that it is owned by Digital Mars and
Symantec.  i wonder what symantec has to do with anything...??

In article <de0j9o$ldt$1 digitaldaemon.com>, Manfred Nowak says...
prefetch <prefetch_member pathlink.com> wrote:

[...]
 holy crap batman, who is driving this boat?? ;-)

This superfast offshore-racer is punched through the waves solely by Walter Bright. Because of the mere mass of tiny fish that is spread all over the water, wanting the boat to race even faster and Walters limited capabilities only fish that is able to mimic a big whale pointing to fast routes through dangerous cliffs and shallows will be recognized. Furthermore everything ever written from that tiny fish becomes immediately a private property of Walters DigitalMars, except expressively stated otherwise. Therefore you just presnted another famous creature to become a private property :-) -manfred

Aug 17 2005
next sibling parent Derek Parnell <derek psych.ward> writes:
On Thu, 18 Aug 2005 00:27:12 +0000 (UTC), prefetch wrote:

 hm.  
 
 so let me translate: 
 
 1. "this is a one man show."
 2. "if you want the one man to do something, you better figure out how to get
 his attention."
 3. "anything you do to help becomes private property of the one man show."
 
 is this correct?

Pretty much, as I understand it.
 i snooped around and couldn't find much licensing info - except the compiler
 comes with a license.txt file that implies that it is owned by Digital Mars and
 Symantec.  i wonder what symantec has to do with anything...??

I believe that Walter had a hand in writing a C++ compiler for Symantec. http://www.walterbright.com/ -- Derek (skype: derek.j.parnell) Melbourne, Australia 18/08/2005 10:31:34 AM
Aug 17 2005
prev sibling parent "Ben Hinkle" <ben.hinkle gmail.com> writes:
"prefetch" <prefetch_member pathlink.com> wrote in message 
news:de0kl0$ml2$1 digitaldaemon.com...
 hm.

 so let me translate:

 1. "this is a one man show."

who responds to email (eventually or usually) and sometimes responds to newsgroup threads.
 2. "if you want the one man to do something, you better figure out how to 
 get
 his attention."

For localized things I've had luck posting about changes, getting feedback, making the changes, making the unittests, updating the doc and emailing Walter with a link to the discussion. For bigger changes I'm at a loss how to proceed.
 3. "anything you do to help becomes private property of the one man show."

 is this correct?

Various parts of phobos have non-Walter (or non-DigitalMars) copyrights. Some of it is in the public domain. I like to put my stuff in the public domain since I don't plan on selling it and I don't care if I lose interest and/or someone else starts selling it. You can also make your own libraries with whatever copyright/license you want.
 i snooped around and couldn't find much licensing info - except the 
 compiler
 comes with a license.txt file that implies that it is owned by Digital 
 Mars and
 Symantec.  i wonder what symantec has to do with anything...??

Different parts come with different licenses. For example poke around in the etc phobos directory and I think none of it is DigitalMars.
Aug 17 2005
prev sibling next sibling parent AJG <AJG_member pathlink.com> writes:
Hi,

Sorry to be the bearer of bad news, but digitalmars.D.bugs *is* the bug
tracking 'system'. Fortunately, Thomas Kühne is helping us with the
'tracking' part.

holy crap batman, who is driving this boat?? ;-) how on earth can you expect to develop a language and runtime library without a bug tracking system? i mean, this is just plain stupid. we all know, there area whole bunch of free, web based bug trackers. don't get me wrong, i've really enjoyed learning about D, and i think it's a stellar language, but COME ON PEOPLE - let's get it together here. somewhere the list of "how to start a super successful grassroots technology" is 'setup a web based bug tracking system so the community can help identify and fix bugs for the primary developer(s).' are my expectations just way to high or something? shouldn't this project reflect the quality of the language concept? i mean, it's a great concept how about we help it not fade into obscurity before it even gets going by building tools (docs, bug tracker, etc.) that can allow the community to grow?

FWIW, I concur with this general sentiment. The PHP community has done some truly outstanding work in this regard. I wish some of that would be emulated for D in the form of better web usage (forum/tracker/docs/feedback/etc.). Cheers, --AJG.
Aug 17 2005
prev sibling parent reply "Ben Hinkle" <ben.hinkle gmail.com> writes:
"prefetch" <prefetch_member pathlink.com> wrote in message 
news:de0fam$ivb$1 digitaldaemon.com...
 In article <1bl4gtbuuy6ek.12qm3vcv0nnxa$.dlg 40tude.net>, Derek Parnell 
 says...

Sorry to be the bearer of bad news, but digitalmars.D.bugs *is* the bug
tracking 'system'. Fortunately, Thomas Kühne is helping us with the
'tracking' part.

holy crap batman, who is driving this boat?? ;-) how on earth can you expect to develop a language and runtime library without a bug tracking system? i mean, this is just plain stupid. we all know, there are a whole bunch of free, web based bug trackers. don't get me wrong, i've really enjoyed learning about D, and i think it's a stellar language, but COME ON PEOPLE - let's get it together here. somewhere on the list of "how to start a super successful grassroots technology" is 'setup a web based bug tracking system so the community can help identify and fix bugs for the primary developer(s).' are my expectations just way to high or something? shouldn't this project reflect the quality of the language concept? i mean, it's a great concept - but how about we help it not fade into obscurity before it even gets going by building tools (docs, bug tracker, etc.) that can allow the community to grow?

One issue with the community fixing bugs is that many of the bugs are in the compiler and those we can't really help with - and besides I think probably only David Friedman (author of gdc) knows compiler stuff well enough to help. About phobos and the doc the community can definitely help but I see several reasons why there hasn't been more progress: 1) the process for making changes isn't well communicated - eg - post proposal, write unittests, update doc, etc 2) the scope of what Walter expects isn't communicated - what is he willing to change? I think he has said phobos is temporary - what's the plan for it? Who should fix the doc? 3) the community likes to do what they want - there are several different libraries that are fairly complete with overlapping features with phobos but different philosophies so little effort is being put into phobos and the doc. Individuals will either submit a new module or submit some bug fixes but it's fairly quiet. Personally I'm waiting on hearing more about item 2 before really helping on phobos.
Aug 18 2005
parent John Reimer <terminal.node gmail.com> writes:
Ben Hinkle wrote:

 
 One issue with the community fixing bugs is that many of the bugs are in the 
 compiler and those we can't really help with - and besides I think probably 
 only David Friedman (author of gdc) knows compiler stuff well enough to 
 help. About phobos and the doc the community can definitely help but I see 
 several reasons why there hasn't been more progress:
 1) the process for making changes isn't well communicated - eg - post 
 proposal, write unittests, update doc, etc
 2) the scope of what Walter expects isn't communicated - what is he willing 
 to change? I think he has said phobos is temporary - what's the plan for it? 
 Who should fix the doc?
 3) the community likes to do what they want - there are several different 
 libraries that are fairly complete with overlapping features with phobos but 
 different philosophies so little effort is being put into phobos and the 
 doc. Individuals will either submit a new module or submit some bug fixes 
 but it's fairly quiet. Personally I'm waiting on hearing more about item 2 
 before really helping on phobos. 
 
 

I should stay out of this, because usually when I open my mouth in this regard, I get myself in trouble. Yet, I wanted to mention what I've observed. It seems that Walter has been very hesitant to include code/bug fixes in phobos from the community if the submitter doesn't clearly indicate the "openness" of the submitted code. He appears to fear future litigation or the submitter demanding code rights later or something to that effect. Perhaps many of the bug fixers are not putting his mind adequately to rest. I'm not quite sure. That or it's just the huge burden he carries, and he's only able to audit so many fixes. -JJR
Aug 19 2005
prev sibling parent "Ben Hinkle" <ben.hinkle gmail.com> writes:
"prefetch" <prefetch_member pathlink.com> wrote in message 
news:de094m$em7$1 digitaldaemon.com...
 thanks ben.  sounds like D is broken in this regard.  exiting main w/o 
 cleaning
 up threads should not result in a segfault.  just to make supre duper 
 sure, i
 wrote a quick test in C using pthreads, and it has no problems.

That's because C doesn't try to run a final GC when main exits - which hoses the threads that could be still running. Mike Swieton posted a fix for src/phobos/internal/dmain2.d that inserted some code after main returned and before the moduleDtor and gc_term calls. I don't actually remember the details, though. See the thread http://www.digitalmars.com/d/archives/digitalmars/D/bugs/156.html. I don't think the code was ever incoporated by Walter - and I don't think he ever commented on the issue in general.
 so, is there a D bug tracker?
 how do i submit a bug?

 i know there is a D bug newsgroup, but i'm assuming that's a discussion 
 forum
 and not an actual bug tracking system (i hope..)

 In article <ddvv95$77p$1 digitaldaemon.com>, Ben Hinkle says...

D doesn't wait for child threads to finish before trying to do a gc and
exit. This causes nasty seg-v's on exit. Get around the problem by making
sure all child threads are done before returning from main().
See for example
http://www.digitalmars.com/d/archives/digitalmars/D/13528.html and
http://www.digitalmars.com/d/archives/digitalmars/D/6419.html and probably
more.


Aug 17 2005