www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Allocating Executable Memory

reply "Maxime Chevalier" <maximechevalierb gmail.com> writes:
New to the D language here. I'm working on a tracing JIT compiler 
and will need to allocate chunks of memory that are marked as 
executable.

Is there a standard way of doing this in D, or do I need to 
directly call into mprotect? If I'm going to be calling mprotect, 
what's the cleanest way to do that, do I need to declare my own 
prototype for the function and its flags, or should I write some 
C code that does the call?

Thanks for your help.
Jul 31 2012
next sibling parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 31-07-2012 19:23, Maxime Chevalier wrote:
 New to the D language here. I'm working on a tracing JIT compiler and
 will need to allocate chunks of memory that are marked as executable.

 Is there a standard way of doing this in D, or do I need to directly
 call into mprotect? If I'm going to be calling mprotect, what's the
 cleanest way to do that, do I need to declare my own prototype for the
 function and its flags, or should I write some C code that does the call?

 Thanks for your help.

There's no standard way to do it. You'll have to either use mprotect or just pass the relevant flags when you call mmap. The functions and constants related to this are all in core.sys.posix.mman. See also: * https://github.com/lycus/mci/blob/master/src/mci/core/memory.d#L71 (and further down) * https://github.com/lycus/mci/blob/master/src/mci/vm/code.d -- Alex Rønne Petersen alex lycus.org http://lycus.org
Jul 31 2012
parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 31-07-2012 19:36, Maxime Chevalier wrote:
 The functions and constants related to this are all in
 core.sys.posix.mman.

 See also:

 * https://github.com/lycus/mci/blob/master/src/mci/core/memory.d#L71
 (and further down)
 * https://github.com/lycus/mci/blob/master/src/mci/vm/code.d

Thanks. Forgive my ignorance though, but does core.sys.posix.sys.mman ship standard with D compilers?

Yep. Anything core.* and std.* is part of druntime and phobos which both ship with any D 2.0 compiler. -- Alex Rønne Petersen alex lycus.org http://lycus.org
Jul 31 2012
next sibling parent =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 31-07-2012 20:06, Maxime Chevalier wrote:
 Yep. Anything core.* and std.* is part of druntime and phobos which
 both ship with any D 2.0 compiler.

Thanks very much for the quick responses. My tracing JIT will likely need to have the compiled code write into data structures of the host VM. This will mean the compiled code holds pointers to host data. How well does the D GC deal with immovable objects. Would it be a problem if I allocated many chunks of immovable data?

Can you clarify what you mean by immovable data? I don't think I follow. In any case, if the compiled code's executable memory regions aren't added as root ranges to D's GC (which is entirely reasonable to not do), then you can keep the data alive by simply keeping pointers to the D GC-managed data in the host environment. D doesn't use a moving collector.
 Would it be preferable to minimize immovable data by having compiled
 code refer to an (immovable) table with pointers to (movable) host data?

 Also, if you don't mind me asking, which D compiler do you guys prefer?
 Which one is most widely supported, most up to date?

DMD: Latest updates, fixes, and enhancements. Very fast compilation. Not that great optimization. GDC: Best optimization and fantastic debug info generation. Somewhat slow at compilation. LDC: Fast compilation and good optimization. Sits somewhere between DMD and GDC. Note that DMD can only target x86 while LDC and GDC (due to using LLVM and GCC respectively) can target many other 32-bit and 64-bit architectures. Also, LDC and GDC tend to lag one release behind DMD sometimes, since they rely on the DMD front end code. -- Alex Rønne Petersen alex lycus.org http://lycus.org
Jul 31 2012
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 07/31/2012 08:06 PM, Maxime Chevalier wrote:
 How well does the D GC deal with immovable objects. Would it be a
 problem if I allocated many chunks of immovable data?

The current GC in druntime never moves data.
Jul 31 2012
prev sibling next sibling parent "Maxime Chevalier" <maximechevalierb gmail.com> writes:
 The functions and constants related to this are all in 
 core.sys.posix.mman.

 See also:

 * 
 https://github.com/lycus/mci/blob/master/src/mci/core/memory.d#L71 
 (and further down)
 * https://github.com/lycus/mci/blob/master/src/mci/vm/code.d

Thanks. Forgive my ignorance though, but does core.sys.posix.sys.mman ship standard with D compilers?
Jul 31 2012
prev sibling next sibling parent Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
--bcaec554d63c0a889504c623be90
Content-Type: text/plain; charset=UTF-8

On Tue, Jul 31, 2012 at 9:36 PM, Maxime Chevalier <
maximechevalierb gmail.com> wrote:

 The functions and constants related to this are all in core.sys.posix.mman.
 See also:

 * https://github.com/lycus/mci/**blob/master/src/mci/core/**memory.d#L71<https://github.com/lycus/mci/blob/master/src/mci/co
e/memory.d#L71>(and further down)
 * https://github.com/lycus/mci/**blob/master/src/mci/vm/code.d<https://github.com/lycus/mci/blob/master/src/mci/vm/code.d>

Thanks. Forgive my ignorance though, but does core.sys.posix.sys.mman ship standard with D compilers?

It's part of druntime. Druntime is D's runtime library, without which D code above version 2.0 can't work. -- Bye, Gor Gyolchanyan. --bcaec554d63c0a889504c623be90 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div class=3D"gmail_quote">On Tue, Jul 31, 2012 at 9:36 PM, Maxime Chevalie= r <span dir=3D"ltr">&lt;<a href=3D"mailto:maximechevalierb gmail.com" targe= t=3D"_blank">maximechevalierb gmail.com</a>&gt;</span> wrote:<br><blockquot= e class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc sol= id;padding-left:1ex"> <div class=3D"im"><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .= 8ex;border-left:1px #ccc solid;padding-left:1ex"> The functions and constants related to this are all in core.sys.posix.mman.= <br> <br> See also:<br> <br> * <a href=3D"https://github.com/lycus/mci/blob/master/src/mci/core/memory.d= #L71" target=3D"_blank">https://github.com/lycus/mci/<u></u>blob/master/src= /mci/core/<u></u>memory.d#L71</a> (and further down)<br> * <a href=3D"https://github.com/lycus/mci/blob/master/src/mci/vm/code.d" ta= rget=3D"_blank">https://github.com/lycus/mci/<u></u>blob/master/src/mci/vm/= code.d</a><br> </blockquote> <br></div> Thanks. Forgive my ignorance though, but does core.sys.posix.sys.mman ship = standard with D compilers?<br> </blockquote></div><br>It&#39;s part of druntime. Druntime is D&#39;s runti= me library, without which D code above version 2.0 can&#39;t work.<br clear= =3D"all"><div><br></div>-- <br>Bye,<br>Gor Gyolchanyan.<br> --bcaec554d63c0a889504c623be90--
Jul 31 2012
prev sibling next sibling parent "Maxime Chevalier" <maximechevalierb gmail.com> writes:
 Yep. Anything core.* and std.* is part of druntime and phobos 
 which both ship with any D 2.0 compiler.

Thanks very much for the quick responses. My tracing JIT will likely need to have the compiled code write into data structures of the host VM. This will mean the compiled code holds pointers to host data. How well does the D GC deal with immovable objects. Would it be a problem if I allocated many chunks of immovable data? Would it be preferable to minimize immovable data by having compiled code refer to an (immovable) table with pointers to (movable) host data? Also, if you don't mind me asking, which D compiler do you guys prefer? Which one is most widely supported, most up to date?
Jul 31 2012
prev sibling next sibling parent reply "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Tuesday, 31 July 2012 at 17:23:08 UTC, Maxime Chevalier wrote:
 New to the D language here. I'm working on a tracing JIT 
 compiler and will need to allocate chunks of memory that are 
 marked as executable.

 Is there a standard way of doing this in D, or do I need to 
 directly call into mprotect? If I'm going to be calling 
 mprotect, what's the cleanest way to do that, do I need to 
 declare my own prototype for the function and its flags, or 
 should I write some C code that does the call?

The x86 chip it's a simple flag that the OS can set. For use with like UPX, the whole section is marked read, write & execute I believe (since it has to expand it first before it can execute the code; That is of course for loaded memory, not allocating...). I would say check their sources, may prove interesting.
Jul 31 2012
parent =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 31-07-2012 20:30, Era Scarecrow wrote:
 On Tuesday, 31 July 2012 at 17:23:08 UTC, Maxime Chevalier wrote:
 New to the D language here. I'm working on a tracing JIT compiler and
 will need to allocate chunks of memory that are marked as executable.

 Is there a standard way of doing this in D, or do I need to directly
 call into mprotect? If I'm going to be calling mprotect, what's the
 cleanest way to do that, do I need to declare my own prototype for the
 function and its flags, or should I write some C code that does the call?

The x86 chip it's a simple flag that the OS can set. For use with like UPX, the whole section is marked read, write & execute I believe (since it has to expand it first before it can execute the code; That is of course for loaded memory, not allocating...). I would say check their sources, may prove interesting.

UPX simply uses VirtualProtect/mprotect. It's the only way you _can_ do this in ring 3 on any OS with sane security. -- Alex Rønne Petersen alex lycus.org http://lycus.org
Jul 31 2012
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Maxime Chevalier:

 How well does the D GC deal with immovable objects. Would it be 
 a problem if I allocated many chunks of immovable data?

The GC supports flags to define memory as movable or immovable, but I think at the moment it performs no memory movements.
 Also, if you don't mind me asking, which D compiler do you guys 
 prefer? Which one is most widely supported, most up to date?

DMD has a back-end that produces less efficient binaries compared to GDC and LDC. But DMD is more up to date (but not a lot), and it's the most supported, it's the reference implementation. DMD works well on Windows too, GDC works on Windows too. I think LDC doesn't work well on Windows. Bye, bearophile
Jul 31 2012
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
bearophile:

 DMD has a back-end that produces less efficient binaries 
 compared
 to GDC and LDC. But DMD is more up to date (but not a lot), and
 it's the most supported, it's the reference implementation. DMD
 works well on Windows too, GDC works on Windows too. I think LDC
 doesn't work well on Windows.

An DMD doesn't support 64 bit on Windows. Bye, bearophile
Jul 31 2012
prev sibling next sibling parent "David Nadlinger" <see klickverbot.at> writes:
On Tuesday, 31 July 2012 at 18:06:28 UTC, Maxime Chevalier wrote:
 How well does the D GC deal with immovable objects. Would it be 
 a problem if I allocated many chunks of immovable data?

D's GC currently doesn't move memory at all, so this should not be a problem. If there is a chance that all references to the object in question in D-managed memory (i.e. stack, GC heap) go away, but the memory is still used, you might need to manually mark the object as live, though (see core.memory.GC.addRoot).
 Also, if you don't mind me asking, which D compiler do you guys 
 prefer? Which one is most widely supported, most up to date?

DMD is the official reference compiler and compiles faster than the others, but is x86-only (32 bit only on Windows), and the performance of the generated code is sometimes significantly worse than with the other compilers. GDC and LDC also use the same frontend as DMD, but it usually takes them a few weeks to catch up after a new version of it is released. LDC currently doesn't work on Windows if your application uses exception handling (which most applications do). I personally use LDC as my default compiler, but I might be biased… ;) David
Jul 31 2012
prev sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Tuesday, 31 July 2012 at 17:23:08 UTC, Maxime Chevalier wrote:
 New to the D language here. I'm working on a tracing JIT 
 compiler and will need to allocate chunks of memory that are 
 marked as executable.

 Is there a standard way of doing this in D, or do I need to 
 directly call into mprotect? If I'm going to be calling 
 mprotect, what's the cleanest way to do that, do I need to 
 declare my own prototype for the function and its flags, or 
 should I write some C code that does the call?

 Thanks for your help.

Hope this helps: http://stackoverflow.com/a/8656294/21501
Jul 31 2012