www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Stack traced exceptions

reply "Maxime Larose" <mlarose broadsoft.com> writes:
Walter,
(or anyone else who can respond)

I finally got the time to investigate built-in stack tracing in the last few
days. FYI, I want to deal with Windows first.

(I'll say bluntly and off the bat that I'm probably not the best person to
tackle this issue. I have no particular knowledge of stack frames, debugging
schemes, symbol tables, omf/coff, etc. I'd whish someone more knowledgeable
than me took up the task and maybe what I say below will sound naive. At any
rate, I believe built-in stack tracing is very important for any new
language. I've made my case in another thread, I will not repeat myself
here.)

So, my initial approach went like this:
- I use MSVC to debug D executables and it usually works fine (templates are
the exception, somtimes the debugger will have a hard time with them -
understandable).
- So, I figured that MSVC probably eat its own dog food, i.e. use
IMAGEHLP.DLL to do its stack walking and symbol referencing.
- The idea is to use StalkWalk() in imagehlp.dll to walk the stack frames.
Digged up old articles on the net about imagehlp.dll and tried to implement
something in phobos.
- There are some undocumented requirements for StackWalk and I need a
CONTEXT. Apparently, a valid CONTEXT can only be obtained when: (1) the
thread is sleeping or (2) an exception is raised.
- Function _d_framehandler in deh.c receives a CONTEXT as a parameter.So, I
hacked Object.d so that a specific (new) exception was raised (w/
RaiseException) in Exception.this(). The _d_framehandler  function traps
that exception and deals with it in a special manner, .i.e. call StackWalk
etc.

Unfortunately, at this point it's not working. The first stack frame I get
back from the library is wrong somehow. I didn't have time to investigate
more yesterday, but I was a bit worried that I'd wasted my time. What if D
(or the digital mars C++ compiler) don't follow the standard stack frame
format. But than how can MSVC debug D apps?  So, yes, D must follow the
standard stack frame format, and MSVC can work with it. However, perhaps
MSVC doesn't use imagehlp.


Again, I may sound naive and ignorant, but I would like to know if imagehlp
_should_ work with D apps. If not, I'd rather know now than engulf countless
hours in a futile attempt. If imagehlp can't work, than that's a whole
different story for me. I'd have to walk the stack myself and that involves
a *lot* more effort (mainly because the learning curve would be as steep as
it gets.)

So:
1. Should imagehlp work with D apps?
2. Perhaps with some tweakings?
3. Anyone (Walter?) that can provide pointers on how stack frames/symbols
are implemented in D, the overall design (ex: this file contains methods
that do this, this other file does this), etc.

The feeling I have is it would be so much simpler for Walter to implement
this... Anyway, I'm not complaining, just trying to save myself a lot of
time (which I don't have - but aren't we all in this same situation? ;).

Thanks,

Max
Apr 27 2005
next sibling parent reply "Ben Hinkle" <bhinkle mathworks.com> writes:
I think it's just done "by hand". On Windows you get the EBP register (or 
EIP) register for the stack frame. That then has the address of the calling 
function and the pointer to the parent stack frame. I'm not sure how to map 
the function address to a string to print. I'd recommend Googling for "Win32 
stack trace" or something for more info. I expect the code to be pretty 
platform-dependent. See for example 
http://www.eptacom.net/pubblicazioni/pub_eng/except.html

"Maxime Larose" <mlarose broadsoft.com> wrote in message 
news:d4ntes$tm5$1 digitaldaemon.com...
 Walter,
 (or anyone else who can respond)

 I finally got the time to investigate built-in stack tracing in the last 
 few
 days. FYI, I want to deal with Windows first.

 (I'll say bluntly and off the bat that I'm probably not the best person to
 tackle this issue. I have no particular knowledge of stack frames, 
 debugging
 schemes, symbol tables, omf/coff, etc. I'd whish someone more 
 knowledgeable
 than me took up the task and maybe what I say below will sound naive. At 
 any
 rate, I believe built-in stack tracing is very important for any new
 language. I've made my case in another thread, I will not repeat myself
 here.)

 So, my initial approach went like this:
 - I use MSVC to debug D executables and it usually works fine (templates 
 are
 the exception, somtimes the debugger will have a hard time with them -
 understandable).
 - So, I figured that MSVC probably eat its own dog food, i.e. use
 IMAGEHLP.DLL to do its stack walking and symbol referencing.
 - The idea is to use StalkWalk() in imagehlp.dll to walk the stack frames.
 Digged up old articles on the net about imagehlp.dll and tried to 
 implement
 something in phobos.
 - There are some undocumented requirements for StackWalk and I need a
 CONTEXT. Apparently, a valid CONTEXT can only be obtained when: (1) the
 thread is sleeping or (2) an exception is raised.
 - Function _d_framehandler in deh.c receives a CONTEXT as a parameter.So, 
 I
 hacked Object.d so that a specific (new) exception was raised (w/
 RaiseException) in Exception.this(). The _d_framehandler  function traps
 that exception and deals with it in a special manner, .i.e. call StackWalk
 etc.

 Unfortunately, at this point it's not working. The first stack frame I get
 back from the library is wrong somehow. I didn't have time to investigate
 more yesterday, but I was a bit worried that I'd wasted my time. What if D
 (or the digital mars C++ compiler) don't follow the standard stack frame
 format. But than how can MSVC debug D apps?  So, yes, D must follow the
 standard stack frame format, and MSVC can work with it. However, perhaps
 MSVC doesn't use imagehlp.


 Again, I may sound naive and ignorant, but I would like to know if 
 imagehlp
 _should_ work with D apps. If not, I'd rather know now than engulf 
 countless
 hours in a futile attempt. If imagehlp can't work, than that's a whole
 different story for me. I'd have to walk the stack myself and that 
 involves
 a *lot* more effort (mainly because the learning curve would be as steep 
 as
 it gets.)

 So:
 1. Should imagehlp work with D apps?
 2. Perhaps with some tweakings?
 3. Anyone (Walter?) that can provide pointers on how stack frames/symbols
 are implemented in D, the overall design (ex: this file contains methods
 that do this, this other file does this), etc.

 The feeling I have is it would be so much simpler for Walter to implement
 this... Anyway, I'm not complaining, just trying to save myself a lot of
 time (which I don't have - but aren't we all in this same situation? ;).

 Thanks,

 Max

 

Apr 27 2005
parent reply brad beveridge <brad nowhere.com> writes:
I might be completely naive here, but why can't we build stack tracking 
functionality into the language?  For example, alter the D compiler to 
have an internal stack of strings (or an index into a string table if 
you are really concerned about speed), and make it so the D compiler 
emits a "push(function_name)" when you call a function and "pop" when 
you return.
That way you get stack tracing for every architechture without having to 
know anything about the backend binary format.

Brad
Apr 27 2005
parent reply "Ben Hinkle" <bhinkle mathworks.com> writes:
"brad beveridge" <brad nowhere.com> wrote in message 
news:d4ords$1tth$1 digitaldaemon.com...
I might be completely naive here, but why can't we build stack tracking 
functionality into the language?  For example, alter the D compiler to have 
an internal stack of strings (or an index into a string table if you are 
really concerned about speed), and make it so the D compiler emits a 
"push(function_name)" when you call a function and "pop" when you return.
 That way you get stack tracing for every architechture without having to 
 know anything about the backend binary format.

 Brad

It would slow down function calls uniformly instead of just paying the price of determining the stack at throw-time. The performance hit would be too big (I assume). Now that I look more closely at the MSDN doc for StackWalk64 I think Maxime is on the right track. MSDN talks about DebugHlp.dll instead of ImageHlp.dll, though.
Apr 27 2005
parent reply Brad Beveridge <brad somewhere.net> writes:
Ben Hinkle wrote:
 It would slow down function calls uniformly instead of just paying the price 
 of determining the stack at throw-time. The performance hit would be too big 
 (I assume).

Now that sounds like premature optimisation :) I think you'd get away with ** function start ** stack_trace[depth++] = function_ID; .... ** function return ** depth--; Where function_ID is a pointer to a string, or something that can be looked up later. Which isn't much overhead. Pros of inbuilt stack tracing 1) Shouldn't be too hard to implement in the compiler (I could be completly wrong here!) 2) Will work on all platforms. 3) Can be enabled/disabled with a compiler switch 4) Don't have to go poking into stack frames, etc at throw time. Cons of inbuilt stack tracing 1) Small uniform overhead for each function called. Brad
Apr 27 2005
parent reply Kevin Bealer <Kevin_member pathlink.com> writes:
In article <d4p1u1$24mm$1 digitaldaemon.com>, Brad Beveridge says...
Ben Hinkle wrote:
 It would slow down function calls uniformly instead of just paying the price 
 of determining the stack at throw-time. The performance hit would be too big 
 (I assume).

Now that sounds like premature optimisation :) I think you'd get away with ** function start ** stack_trace[depth++] = function_ID; .... ** function return ** depth--; Where function_ID is a pointer to a string, or something that can be looked up later. Which isn't much overhead. Pros of inbuilt stack tracing 1) Shouldn't be too hard to implement in the compiler (I could be completly wrong here!) 2) Will work on all platforms. 3) Can be enabled/disabled with a compiler switch 4) Don't have to go poking into stack frames, etc at throw time. Cons of inbuilt stack tracing 1) Small uniform overhead for each function called. Brad

While this isn't too much overhead for heavy functions, I think for getters and setters it would swamp you. But, the functions always return where they are supposed to, right? That is due to the fact that the "stack" already does what you are proposing, only its idea of a "function id" is the function pointer. I am guessing (from a POV of no facts) that the complex part of this would be trying to work out which "inlined" function you are in based on ranges of insns. This can be done if you have debug data (I think) because C++ programs in gdb show this. Funny thing is, the code seems to go in-out-in of inlined and optimized methods because it all gets mixed and reordered by the optimizer. If you want to do this portably, though, a better way that inserting "push/pop" on every call, might be (for the compiler) to insert something like: try { .. function contents } catch(TracedException e) { e.push_frame_name("function name"); throw e; } ..around every function body. Then you only pay the piper in the sense that you have increased the bloat factor, and neutered the inlining phase of the optimizer. Of course if you can (at compile time) determine that this function can't/won't throw a traced exception, then you can omit the try{}catch{}. Another downside is that you only get stack tracing from throw-point to catch-point, but I suspect if you want a stack trace, it is probably thrown "all the way" to main or further, right? Kevin
Apr 28 2005
parent reply "Maxime Larose" <mlarose broadsoft.com> writes:
Pushing function names on the stack and the try-catch method have one
important drawback: you lose granularity. You know in which function the
exception occured, but you don't know where inside the function. The only
way to know where exactly the exception was thrown is by inspecting EIP/EBP
and walking the stack from there.

As Ben said, walking the stack is not that difficult really. I was mentally
preparing myself to do it manually, but in the process of stepping through
my imagehlp code, I found out that some structures were badly filled by
Windows. I now fill them myself with some tricks. Anyways, to make a long
story short, I made a lot of progress yesterday and the overall design will
be a lot more elegant.

I now have a stack trace that lists:
Method name 1 + displacement
Method name 2 + displacement
Method name 3 + displacement
...

My idea is to not limit such stack traces to instances where Exception is
created manually, but also to cases where you have null pointers and other
such "system" exceptions. (BTW Kevin, that would eliminate the possibility
for the compiler to eliminate the try-catch at compile time, since any code
can throw). I little nudging in deh.c will do the trick.

The problem now is the displacement. I don't want a displacement, I want a
line number. I had trouble making it work though as the function that
resolves line numbers on Windows seems flaky.

I'm going to spend more time on it, but Ben mentionned something good:
debughlp vs imagehlp. Debughlp is only available on Windows XP and Windows
2000 professional. I assumed that was not acceptable, but now I'm
reconsidering. I guess everyone is on Windows XP or 2000 by now (time
flies!).

So, I will use debughlp and support AMD64 out of the box.

Thanks for your comments,

Max




"Kevin Bealer" <Kevin_member pathlink.com> wrote in message
news:d4q251$l5$1 digitaldaemon.com...
 In article <d4p1u1$24mm$1 digitaldaemon.com>, Brad Beveridge says...
Ben Hinkle wrote:
 It would slow down function calls uniformly instead of just paying the



 of determining the stack at throw-time. The performance hit would be



 (I assume).

Now that sounds like premature optimisation :) I think you'd get away with ** function start ** stack_trace[depth++] = function_ID; .... ** function return ** depth--; Where function_ID is a pointer to a string, or something that can be looked up later. Which isn't much overhead. Pros of inbuilt stack tracing 1) Shouldn't be too hard to implement in the compiler (I could be completly wrong here!) 2) Will work on all platforms. 3) Can be enabled/disabled with a compiler switch 4) Don't have to go poking into stack frames, etc at throw time. Cons of inbuilt stack tracing 1) Small uniform overhead for each function called. Brad

While this isn't too much overhead for heavy functions, I think for

 setters it would swamp you.

 But, the functions always return where they are supposed to, right?  That

 to the fact that the "stack" already does what you are proposing, only its

 of a "function id" is the function pointer.

 I am guessing (from a POV of no facts) that the complex part of this would

 trying to work out which "inlined" function you are in based on ranges of

 This can be done if you have debug data (I think) because C++ programs in

 show this.

 Funny thing is, the code seems to go in-out-in of inlined and optimized

 because it all gets mixed and reordered by the optimizer.

 If you want to do this portably, though, a better way that inserting

 on every call, might be (for the compiler) to insert something like:

 try {
 .. function contents
 }
 catch(TracedException e) {
 e.push_frame_name("function name");
 throw e;
 }

 ..around every function body.  Then you only pay the piper in the sense

 you have increased the bloat factor, and neutered the inlining phase of

 optimizer.  Of course if you can (at compile time) determine that this

 can't/won't throw a traced exception, then you can omit the try{}catch{}.

 Another downside is that you only get stack tracing from throw-point to
 catch-point, but I suspect if you want a stack trace, it is probably

 the way" to main or further, right?

 Kevin

Apr 28 2005
parent reply "Ben Hinkle" <ben.hinkle gmail.com> writes:
 I'm going to spend more time on it, but Ben mentionned something good:
 debughlp vs imagehlp. Debughlp is only available on Windows XP and Windows
 2000 professional. I assumed that was not acceptable, but now I'm
 reconsidering. I guess everyone is on Windows XP or 2000 by now (time
 flies!).

 So, I will use debughlp and support AMD64 out of the box.

I searched for StackWalk on MSDN and didn't find anything except the debughlp version. I notice, though, that the include headers in dm/include/win32 only talk about StackWalk and not StackWalk64. I don't know which is better to use - maybe the 64 version would be fine behind a version(X86_64) or something.
Apr 28 2005
parent "Maxime Larose" <mlarose broadsoft.com> writes:
The StalkWalk64 version deals with 32-bit apps also.

My statement regarding StalkWalk64 as only being available under
Windows2000+ is false. You can have it for previous versions as well if you
get your hands on the dbghelp.dll redistributable. So, that's settled!

I don't want to jump the gun too much, but if all goes well, we will have
stack traced D in a few days/weeks. For windows that is. For Linux, I
unfortunately don't have it installed at home. I was considering getting a
unix server up for other reasons, but I pretty much decided against that for
the moment. So, if anyone wants to step in for linux...





"Ben Hinkle" <ben.hinkle gmail.com> wrote in message
news:d4qj84$kjr$1 digitaldaemon.com...
 I'm going to spend more time on it, but Ben mentionned something good:
 debughlp vs imagehlp. Debughlp is only available on Windows XP and


 2000 professional. I assumed that was not acceptable, but now I'm
 reconsidering. I guess everyone is on Windows XP or 2000 by now (time
 flies!).

 So, I will use debughlp and support AMD64 out of the box.

I searched for StackWalk on MSDN and didn't find anything except the debughlp version. I notice, though, that the include headers in dm/include/win32 only talk about StackWalk and not StackWalk64. I don't

 which is better to use - maybe the 64 version would be fine behind a
 version(X86_64) or something.

Apr 28 2005
prev sibling parent reply Sean Kelly <sean f4.ca> writes:
In article <d4ntes$tm5$1 digitaldaemon.com>, Maxime Larose says...
- There are some undocumented requirements for StackWalk and I need a
CONTEXT. Apparently, a valid CONTEXT can only be obtained when: (1) the
thread is sleeping or (2) an exception is raised.

Look at std.thread and/or internal.gc.gcx. One of those grabs a CONTEXT structure in Windows builds. This is used to get stack pointers for garbage collection. Sean
Apr 30 2005
parent "Maxime Larose" <mlarose broadsoft.com> writes:
"Sean Kelly" <sean f4.ca> wrote in message
news:d50f1t$s4m$1 digitaldaemon.com...
 In article <d4ntes$tm5$1 digitaldaemon.com>, Maxime Larose says...
- There are some undocumented requirements for StackWalk and I need a
CONTEXT. Apparently, a valid CONTEXT can only be obtained when: (1) the
thread is sleeping or (2) an exception is raised.

Look at std.thread and/or internal.gc.gcx. One of those grabs a CONTEXT structure in Windows builds. This is used to get stack pointers for

 collection.


 Sean

I checked std.thread: nothing. In internal.gc.gcx, the CONTEXT is gotten with GetThreadContext. Problem there is that the thread must be stopped for the CONTEXT to be valid...
May 02 2005