www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Unexpected behavior when misusing inline assembly

reply "Noah" <nesanter knox.edu> writes:
When running the following code:

__gshared void* return_ptr;
__gshared void* injected_fn = &fn;

void main() {
     buffer(7);
     printf("End main\n");
}

void buffer() {
     test(x);
     printf("End buffer\n");
}

void test() {
     printf("This is a test!\n");
     inject();
     printf("End of the test!\n");
}

void fn() {
     printf("Hello, world!\n");
     asm {
         mov RAX, return_ptr;
         mov [RBP+8], RAX;
     }
}

void inject() {
     asm {
         naked;
         push RAX;
         mov RAX, [RBP+8];
         mov return_ptr, RAX;
         mov RAX, injected_fn;
         mov [RBP+8], RAX;
         pop RAX;
         ret;
     }
}

The program behaves as expected, that is, the program prints
This is a test!
End of the test!
Hello, world!
End buffer
End main

However, if I call test directly from main, it results in a 
segfault.
If I call inject directly from main, it works.
And, oddly, if I add any amount of inline assembler to main, and 
call test there, it works.

Could anyone explain to me what's going on?
Nov 12 2013
parent reply "Noah" <nesanter knox.edu> writes:
Just realized I forgot to delete the 7 and the x from inside the
function calls.  Question is the same, though.

On Tuesday, 12 November 2013 at 18:22:46 UTC, Noah wrote:
 When running the following code:

 __gshared void* return_ptr;
 __gshared void* injected_fn = &fn;

 void main() {
     buffer(7);
     printf("End main\n");
 }

 void buffer() {
     test(x);
     printf("End buffer\n");
 }

 void test() {
     printf("This is a test!\n");
     inject();
     printf("End of the test!\n");
 }

 void fn() {
     printf("Hello, world!\n");
     asm {
         mov RAX, return_ptr;
         mov [RBP+8], RAX;
     }
 }

 void inject() {
     asm {
         naked;
         push RAX;
         mov RAX, [RBP+8];
         mov return_ptr, RAX;
         mov RAX, injected_fn;
         mov [RBP+8], RAX;
         pop RAX;
         ret;
     }
 }

 The program behaves as expected, that is, the program prints
 This is a test!
 End of the test!
 Hello, world!
 End buffer
 End main

 However, if I call test directly from main, it results in a 
 segfault.
 If I call inject directly from main, it works.
 And, oddly, if I add any amount of inline assembler to main, 
 and call test there, it works.

 Could anyone explain to me what's going on?
Nov 12 2013
parent "Noah" <nesanter knox.edu> writes:
Of course after I post this I realize the answer.  The function 
only works when the surrounding function has at least one 
argument.  Probably has something to do with the way the compiler 
emits no-argument functions, I'll have to look at some 
disassembly.
Nov 12 2013