www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - thread.d "Unable to load thread context"

reply Benjamin Thaut <code benjamin-thaut.de> writes:
I'm my current project, which is a relatime 3d space shooter I'm using a 
lot of C libraries (SDL, OpenGL, OpenAL, SDL_Image, Assimp, etc.)
As the gc would take to much time when it runs only once in a while I'm 
performing a full collect after a frame has been rendered.
Now once in a while, sometimes after 30 seconds, sometimes after 20 
minutes, the D runtime throws the "Unable to load thread context" 
exception. My debugger tells me that the main thread is currenlty 
performing a full collection when that happens.
A few other threads are waiting for single or multiple objects (on 
windows that is). A OpenGL thread runs the 
ntdll!ZwQueryInformationThread (I'm using a nvidia card so it's the 
nvidia OpenGL implementation) and one of my threads waits for a 
semaphore with a tryWait.

My application runs 3 thraeds.
-A main thread where the renderer runs in. This thread makes all the 
OpenGL calls.
-A extractor thread that extracts relevant data for the renderer from 
the game thread.
-The game thread that actually performs the game simulation. This thread 
makes calls to the OpenAL api.

Now I'm wondering why this happens. Is this connected to OpenGL?
Is it not a good idea to call the gc collect manually?
Isthe thread that is waiting for the semaphore the problem?

Any help would be appreciated.
-- 
Kind Regards
Benjamin Thaut
Jun 28 2011
next sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
Have you tried running the latest runtime from git?  I fixed a rather nasty t=
hreading bug since the last release that could be related.=20

Sent from my iPhone

On Jun 28, 2011, at 6:47 AM, Benjamin Thaut <code benjamin-thaut.de> wrote:

 I'm my current project, which is a relatime 3d space shooter I'm using a l=

 As the gc would take to much time when it runs only once in a while I'm pe=

 Now once in a while, sometimes after 30 seconds, sometimes after 20 minute=

bugger tells me that the main thread is currenlty performing a full collecti= on when that happens.
 A few other threads are waiting for single or multiple objects (on windows=

g a nvidia card so it's the nvidia OpenGL implementation) and one of my thre= ads waits for a semaphore with a tryWait.
=20
 My application runs 3 thraeds.
 -A main thread where the renderer runs in. This thread makes all the OpenG=

 -A extractor thread that extracts relevant data for the renderer from the g=

 -The game thread that actually performs the game simulation. This thread m=

=20
 Now I'm wondering why this happens. Is this connected to OpenGL?
 Is it not a good idea to call the gc collect manually?
 Isthe thread that is waiting for the semaphore the problem?
=20
 Any help would be appreciated.
 --=20
 Kind Regards
 Benjamin Thaut

Jun 28 2011
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 28.06.2011 18:09, schrieb Sean Kelly:
 Have you tried running the latest runtime from git?  I fixed a rather nasty
threading bug since the last release that could be related.

 Sent from my iPhone

 On Jun 28, 2011, at 6:47 AM, Benjamin Thaut<code benjamin-thaut.de>  wrote:

 I'm my current project, which is a relatime 3d space shooter I'm using a lot
of C libraries (SDL, OpenGL, OpenAL, SDL_Image, Assimp, etc.)
 As the gc would take to much time when it runs only once in a while I'm
performing a full collect after a frame has been rendered.
 Now once in a while, sometimes after 30 seconds, sometimes after 20 minutes,
the D runtime throws the "Unable to load thread context" exception. My debugger
tells me that the main thread is currenlty performing a full collection when
that happens.
 A few other threads are waiting for single or multiple objects (on windows
that is). A OpenGL thread runs the ntdll!ZwQueryInformationThread (I'm using a
nvidia card so it's the nvidia OpenGL implementation) and one of my threads
waits for a semaphore with a tryWait.

 My application runs 3 thraeds.
 -A main thread where the renderer runs in. This thread makes all the OpenGL
calls.
 -A extractor thread that extracts relevant data for the renderer from the game
thread.
 -The game thread that actually performs the game simulation. This thread makes
calls to the OpenAL api.

 Now I'm wondering why this happens. Is this connected to OpenGL?
 Is it not a good idea to call the gc collect manually?
 Isthe thread that is waiting for the semaphore the problem?

 Any help would be appreciated.
 --
 Kind Regards
 Benjamin Thaut


I just build dmd, druntime and phobos fromt the latest git revision (as druntime does not build with dmd 2.053). The game does build with it, but as soon as it starts a exception gets thrown in the GC which does not get cought by any of my catch blocks. Maybe I did something wrong building dmd, is there a guide somewhere? -- Kind Regards Benjamin Thaut
Jun 28 2011
next sibling parent reply "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Benjamin Thaut" <code benjamin-thaut.de> wrote in message 
news:iud7qf$16jf$1 digitalmars.com...
 I just build dmd, druntime and phobos fromt the latest git revision (as 
 druntime does not build with dmd 2.053). The game does build with it, but 
 as soon as it starts a exception gets thrown in the GC which does not get 
 cought by any of my catch blocks.
 Maybe I did something wrong building dmd, is there a guide somewhere?

Recent changes have made allocating memory from the GC while a collection is running (eg inside a class destructor) throw an exception. Is it possible your code is doing this?
Jun 29 2011
next sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 29.06.2011 09:40, schrieb Daniel Murphy:
 "Benjamin Thaut"<code benjamin-thaut.de>  wrote in message
 news:iud7qf$16jf$1 digitalmars.com...
 I just build dmd, druntime and phobos fromt the latest git revision (as
 druntime does not build with dmd 2.053). The game does build with it, but
 as soon as it starts a exception gets thrown in the GC which does not get
 cought by any of my catch blocks.
 Maybe I did something wrong building dmd, is there a guide somewhere?

Recent changes have made allocating memory from the GC while a collection is running (eg inside a class destructor) throw an exception. Is it possible your code is doing this?

parser I'm still using. Thanks for the tip. -- Kind Regards Benjamin Thaut
Jun 29 2011
parent "Daniel Murphy" <yebblies nospamgmail.com> writes:
"Benjamin Thaut" <code benjamin-thaut.de> wrote in message 
news:iuf6ol$1qrd$1 digitalmars.com...
 It actually was a manual delete inside a destructor from some old D1 xml 
 parser I'm still using.
 Thanks for the tip.

Glad to be helpful.
Jun 29 2011
prev sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 29.06.2011 17:52, schrieb Sean Kelly:
 I really want to fix this--the current patch is frustrating to deal with--but
I don't have the time yet.

 Sent from my iPhone

 On Jun 29, 2011, at 12:40 AM, "Daniel Murphy"<yebblies nospamgmail.com>  wrote:

 "Benjamin Thaut"<code benjamin-thaut.de>  wrote in message
 news:iud7qf$16jf$1 digitalmars.com...
 I just build dmd, druntime and phobos fromt the latest git revision (as
 druntime does not build with dmd 2.053). The game does build with it, but
 as soon as it starts a exception gets thrown in the GC which does not get
 cought by any of my catch blocks.
 Maybe I did something wrong building dmd, is there a guide somewhere?

Recent changes have made allocating memory from the GC while a collection is running (eg inside a class destructor) throw an exception. Is it possible your code is doing this?


(change m to __gshared in slock) the issue still exsits. After a while the game will stop with "unable to load thread context". When compiling with the current git revision, it no longer throws this exception but enters _on_out_of_memory and throws a exception from there (although I don't know where the call is coming from because the debugger can't fully resolve the stack). I'm pretty shure though that I'm not out of memory when this happens (windowns task manager shows that there is 1 gig of memory free), and I also checked all desturctors to not allocate memory. So I don't have any clue where that could be coming from. The out of memory exception occurs during a full collect. -- Kind Regards Benjamin Thaut
Jun 29 2011
next sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 30.06.2011 06:45, schrieb Sean Kelly:
 On Jun 29, 2011, at 10:00 AM, Benjamin Thaut wrote:

 Am 29.06.2011 17:52, schrieb Sean Kelly:
 I really want to fix this--the current patch is frustrating to deal with--but
I don't have the time yet.

 Sent from my iPhone

 On Jun 29, 2011, at 12:40 AM, "Daniel Murphy"<yebblies nospamgmail.com>  
wrote:

 "Benjamin Thaut"<code benjamin-thaut.de>   wrote in message
 news:iud7qf$16jf$1 digitalmars.com...
 I just build dmd, druntime and phobos fromt the latest git revision (as
 druntime does not build with dmd 2.053). The game does build with it, but
 as soon as it starts a exception gets thrown in the GC which does not get
 cought by any of my catch blocks.
 Maybe I did something wrong building dmd, is there a guide somewhere?

Recent changes have made allocating memory from the GC while a collection is running (eg inside a class destructor) throw an exception. Is it possible your code is doing this?



Just for kicks, rewrite the GetThreadContext call as this and see if it works in 2.053: for( int i = 0; !GetThreadContext( t.m_hndl,&context ); i++ ) { if( i> 99 ) throw new ThreadException( "Unable to load thread context" ); Thread.yield(); }

The for loop seems to have fixed the issue, at least I'm not able to reproduce it easily any more. I'm going to be able to tell you tomorrow if the issue is fixed completely or not. Does any access to the GC cause a Exception to be thrown? I mean even a removeRange call? -- Kind Regards Benjamin Thaut
Jun 30 2011
parent reply "Eric Poggel (JoeCoder)" <dnewsgroup2 yage3d.net> writes:
On 6/30/2011 3:05 PM, Sean Kelly wrote:
 On Jun 30, 2011, at 2:40 AM, Benjamin Thaut wrote:

 Am 30.06.2011 06:45, schrieb Sean Kelly:
 On Jun 29, 2011, at 10:00 AM, Benjamin Thaut wrote:
 If I manually merge your fix into the thread.d of the 2.053 runtime (change m
to __gshared in slock) the issue still exsits. After a while the game will stop
with "unable to load thread context".

Just for kicks, rewrite the GetThreadContext call as this and see if it works in 2.053: for( int i = 0; !GetThreadContext( t.m_hndl,&context ); i++ ) { if( i> 99 ) throw new ThreadException( "Unable to load thread context" ); Thread.yield(); }

The for loop seems to have fixed the issue, at least I'm not able to reproduce it easily any more. I'm going to be able to tell you tomorrow if the issue is fixed completely or not.

Cool. I've applied the change to druntime.
 Does any access to the GC cause a Exception to be thrown? I mean even a
removeRange call?

malloc, realloc, extend, reserve, addRoot, and addRange. I think that's it.

This makes me wonder how many other such hacks like this are in DRuntime. If this problem happens once every 30 minutes, then I suppose this decreases it to once every 30^100 minutes (which is older than the universe).
Jun 30 2011
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 01.07.2011 03:51, schrieb Sean Kelly:
 This is the only one I can think of, and it was added based on a suggestion
following a report of a similar bug elsewhere. It sounds like in some weird
circumstances, SuspendThread may return before the thread is suspended. There's
no way to determine this and I'd call it a kernel bug if true, and the only
real option is to spin on GetThreadContext for a bit before giving up and
halting the app.

 Sent from my iPhone

 On Jun 30, 2011, at 4:31 PM, "Eric Poggel (JoeCoder)"<dnewsgroup2 yage3d.net> 
wrote:

 On 6/30/2011 3:05 PM, Sean Kelly wrote:
 On Jun 30, 2011, at 2:40 AM, Benjamin Thaut wrote:

 Am 30.06.2011 06:45, schrieb Sean Kelly:
 On Jun 29, 2011, at 10:00 AM, Benjamin Thaut wrote:
 If I manually merge your fix into the thread.d of the 2.053 runtime (change m
to __gshared in slock) the issue still exsits. After a while the game will stop
with "unable to load thread context".

Just for kicks, rewrite the GetThreadContext call as this and see if it works in 2.053: for( int i = 0; !GetThreadContext( t.m_hndl,&context ); i++ ) { if( i> 99 ) throw new ThreadException( "Unable to load thread context" ); Thread.yield(); }

The for loop seems to have fixed the issue, at least I'm not able to reproduce it easily any more. I'm going to be able to tell you tomorrow if the issue is fixed completely or not.

Cool. I've applied the change to druntime.
 Does any access to the GC cause a Exception to be thrown? I mean even a
removeRange call?

malloc, realloc, extend, reserve, addRoot, and addRange. I think that's it.

This makes me wonder how many other such hacks like this are in DRuntime. If this problem happens once every 30 minutes, then I suppose this decreases it to once every 30^100 minutes (which is older than the universe).


The loop unfortunately didn't fix the problem. It just became less frequent. I even tried adding additional Sleep time to the loop without success. -- Kind Regards Benjamin Thaut
Jul 01 2011
parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 01.07.2011 15:00, schrieb Sean Kelly:
 Then something else must be going on. This is Windows XP/7 and not WINE, right?

 Sent from my iPhone

Yes this is has been tested on 4 different Windows 7 machines. My game is cross platform, and on Linux it runs absolutly stable. (Without wine, directly on linux) If you want I can provide you with the sourcecode and everything else needed, or I can test proposed fixes if you have any. -- Kind Regards Benjamin Thaut
Jul 01 2011
prev sibling parent Sean Kelly <sean invisibleduck.org> writes:
On Jun 30, 2011, at 2:40 AM, Benjamin Thaut wrote:

 Am 30.06.2011 06:45, schrieb Sean Kelly:
 On Jun 29, 2011, at 10:00 AM, Benjamin Thaut wrote:
 If I manually merge your fix into the thread.d of the 2.053 runtime =



the game will stop with "unable to load thread context".
=20
 Just for kicks, rewrite the GetThreadContext call as this and see if =


=20
             for( int i =3D 0; !GetThreadContext( t.m_hndl,&context ); =


             {
                 if( i>  99 )
                     throw new ThreadException( "Unable to load thread =


                 Thread.yield();
             }
=20

The for loop seems to have fixed the issue, at least I'm not able to =

if the issue is fixed completely or not. Cool. I've applied the change to druntime.
 Does any access to the GC cause a Exception to be thrown? I mean even =

malloc, realloc, extend, reserve, addRoot, and addRange. I think that's = it.=
Jun 30 2011
prev sibling next sibling parent Sean Kelly <sean invisibleduck.org> writes:
On Jun 29, 2011, at 10:00 AM, Benjamin Thaut wrote:

 Am 29.06.2011 17:52, schrieb Sean Kelly:
 I really want to fix this--the current patch is frustrating to deal =


=20
 Sent from my iPhone
=20
 On Jun 29, 2011, at 12:40 AM, "Daniel =


=20
 "Benjamin Thaut"<code benjamin-thaut.de>  wrote in message
 news:iud7qf$16jf$1 digitalmars.com...
 I just build dmd, druntime and phobos fromt the latest git revision =




 druntime does not build with dmd 2.053). The game does build with =




 as soon as it starts a exception gets thrown in the GC which does =




 cought by any of my catch blocks.
 Maybe I did something wrong building dmd, is there a guide =




=20

Recent changes have made allocating memory from the GC while a =



 running (eg inside a class destructor) throw an exception.  Is it =



 your code is doing this?
=20
=20



the game will stop with "unable to load thread context". It seems like there may some weirdness with GetThreadContext. See for = example: = http://stackoverflow.com/questions/3444190/windows-suspendthread-doesnt-ge= tthreadcontext-fails I'd have to do more digging to see what to try.
 When compiling with the current git revision, it no longer throws this =

(although I don't know where the call is coming from because the = debugger can't fully resolve the stack). I'm pretty shure though that = I'm not out of memory when this happens (windowns task manager shows = that there is 1 gig of memory free), and I also checked all desturctors = to not allocate memory. So I don't have any clue where that could be = coming from.
 The out of memory exception occurs during a full collect.

That part is definitely related to the "throw OOME when a finalizer = accesses the GC during a collection." I plan to fix this, but it isn't = trivial in the current GC and I'm revisiting using Leandro's GC instead. = I'm also going to be out of touch for a while, so I likely won't get it = done soon.=
Jun 29 2011
prev sibling next sibling parent Sean Kelly <sean invisibleduck.org> writes:
On Jun 29, 2011, at 10:00 AM, Benjamin Thaut wrote:

 Am 29.06.2011 17:52, schrieb Sean Kelly:
 I really want to fix this--the current patch is frustrating to deal =


=20
 Sent from my iPhone
=20
 On Jun 29, 2011, at 12:40 AM, "Daniel =


=20
 "Benjamin Thaut"<code benjamin-thaut.de>  wrote in message
 news:iud7qf$16jf$1 digitalmars.com...
 I just build dmd, druntime and phobos fromt the latest git revision =




 druntime does not build with dmd 2.053). The game does build with =




 as soon as it starts a exception gets thrown in the GC which does =




 cought by any of my catch blocks.
 Maybe I did something wrong building dmd, is there a guide =




=20

Recent changes have made allocating memory from the GC while a =



 running (eg inside a class destructor) throw an exception.  Is it =



 your code is doing this?
=20
=20



the game will stop with "unable to load thread context". Just for kicks, rewrite the GetThreadContext call as this and see if it = works in 2.053: for( int i =3D 0; !GetThreadContext( t.m_hndl, &context ); = i++ ) { if( i > 99 ) throw new ThreadException( "Unable to load thread = context" ); Thread.yield(); }
Jun 29 2011
prev sibling parent Sean Kelly <sean invisibleduck.org> writes:
On Jul 1, 2011, at 7:46 AM, Benjamin Thaut wrote:

 Am 01.07.2011 15:00, schrieb Sean Kelly:
 Then something else must be going on. This is Windows XP/7 and not =


=20
 Yes this is has been tested on 4 different Windows 7 machines.
=20
 My game is cross platform, and on Linux it runs absolutly stable. =

=20
 If you want I can provide you with the sourcecode and everything else =

Can you run it in a debugger and send me stack traces of all the threads = when GetThreadContext fails?=
Jul 01 2011
prev sibling next sibling parent Sean Kelly <sean invisibleduck.org> writes:
I really want to fix this--the current patch is frustrating to deal with--bu=
t I don't have the time yet.=20

Sent from my iPhone

On Jun 29, 2011, at 12:40 AM, "Daniel Murphy" <yebblies nospamgmail.com> wro=
te:

 "Benjamin Thaut" <code benjamin-thaut.de> wrote in message=20
 news:iud7qf$16jf$1 digitalmars.com...
 I just build dmd, druntime and phobos fromt the latest git revision (as=20=


 druntime does not build with dmd 2.053). The game does build with it, but=


 as soon as it starts a exception gets thrown in the GC which does not get=


 cought by any of my catch blocks.
 Maybe I did something wrong building dmd, is there a guide somewhere?
=20

Recent changes have made allocating memory from the GC while a collection i=

 running (eg inside a class destructor) throw an exception.  Is it possible=

 your code is doing this?=20
=20
=20

Jun 29 2011
prev sibling next sibling parent Sean Kelly <sean invisibleduck.org> writes:
This is the only one I can think of, and it was added based on a suggestion f=
ollowing a report of a similar bug elsewhere. It sounds like in some weird c=
ircumstances, SuspendThread may return before the thread is suspended. There=
's no way to determine this and I'd call it a kernel bug if true, and the on=
ly real option is to spin on GetThreadContext for a bit before giving up and=
 halting the app.=20

Sent from my iPhone

On Jun 30, 2011, at 4:31 PM, "Eric Poggel (JoeCoder)" <dnewsgroup2 yage3d.ne=
t> wrote:

 On 6/30/2011 3:05 PM, Sean Kelly wrote:
 On Jun 30, 2011, at 2:40 AM, Benjamin Thaut wrote:
=20
 Am 30.06.2011 06:45, schrieb Sean Kelly:
 On Jun 29, 2011, at 10:00 AM, Benjamin Thaut wrote:
 If I manually merge your fix into the thread.d of the 2.053 runtime (c=





e will stop with "unable to load thread context".
=20
 Just for kicks, rewrite the GetThreadContext call as this and see if it=




=20
             for( int i =3D 0; !GetThreadContext( t.m_hndl,&context ); i=




             {
                 if( i>   99 )
                     throw new ThreadException( "Unable to load thread c=




                 Thread.yield();
             }
=20

The for loop seems to have fixed the issue, at least I'm not able to rep=



ssue is fixed completely or not.
=20
 Cool.  I've applied the change to druntime.
=20
 Does any access to the GC cause a Exception to be thrown? I mean even a r=



=20
 malloc, realloc, extend, reserve, addRoot, and addRange.  I think that's i=


=20
 This makes me wonder how many other such hacks like this are in DRuntime. =

s it to once every 30^100 minutes (which is older than the universe).
Jun 30 2011
prev sibling parent Sean Kelly <sean invisibleduck.org> writes:
Then something else must be going on. This is Windows XP/7 and not WINE, rig=
ht?

Sent from my iPhone

On Jul 1, 2011, at 2:19 AM, Benjamin Thaut <code benjamin-thaut.de> wrote:

 Am 01.07.2011 03:51, schrieb Sean Kelly:
 This is the only one I can think of, and it was added based on a suggesti=


rd circumstances, SuspendThread may return before the thread is suspended. T= here's no way to determine this and I'd call it a kernel bug if true, and th= e only real option is to spin on GetThreadContext for a bit before giving up= and halting the app.
=20
 Sent from my iPhone
=20
 On Jun 30, 2011, at 4:31 PM, "Eric Poggel (JoeCoder)"<dnewsgroup2 yage3d.=


=20
 On 6/30/2011 3:05 PM, Sean Kelly wrote:
 On Jun 30, 2011, at 2:40 AM, Benjamin Thaut wrote:
=20
 Am 30.06.2011 06:45, schrieb Sean Kelly:
 On Jun 29, 2011, at 10:00 AM, Benjamin Thaut wrote:
 If I manually merge your fix into the thread.d of the 2.053 runtime (=







me will stop with "unable to load thread context".
=20
 Just for kicks, rewrite the GetThreadContext call as this and see if i=






=20
             for( int i =3D 0; !GetThreadContext( t.m_hndl,&context );=






             {
                 if( i>    99 )
                     throw new ThreadException( "Unable to load thread=






                 Thread.yield();
             }
=20

The for loop seems to have fixed the issue, at least I'm not able to r=





e issue is fixed completely or not.
=20
 Cool.  I've applied the change to druntime.
=20
 Does any access to the GC cause a Exception to be thrown? I mean even a=





=20
 malloc, realloc, extend, reserve, addRoot, and addRange.  I think that'=




=20
 This makes me wonder how many other such hacks like this are in DRuntime=



ses it to once every 30^100 minutes (which is older than the universe).
=20
 The loop unfortunately didn't fix the problem. It just became less frequen=

=20
 --=20
 Kind Regards
 Benjamin Thaut

Jul 01 2011