digitalmars.D - Allocating Executable Memory
- "Maxime Chevalier" <maximechevalierb gmail.com> Jul 31 2012
- =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> Jul 31 2012
- =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> Jul 31 2012
- =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> Jul 31 2012
- Timon Gehr <timon.gehr gmx.ch> Jul 31 2012
- "Maxime Chevalier" <maximechevalierb gmail.com> Jul 31 2012
- Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> Jul 31 2012
- "Maxime Chevalier" <maximechevalierb gmail.com> Jul 31 2012
- "Era Scarecrow" <rtcvb32 yahoo.com> Jul 31 2012
- "bearophile" <bearophileHUGS lycos.com> Jul 31 2012
- "bearophile" <bearophileHUGS lycos.com> Jul 31 2012
- "David Nadlinger" <see klickverbot.at> Jul 31 2012
- "Vladimir Panteleev" <vladimir thecybershadow.net> Jul 31 2012
New to the D language here. I'm working on a tracing JIT compiler and will need to allocate chunks of memory that are marked as executable. Is there a standard way of doing this in D, or do I need to directly call into mprotect? If I'm going to be calling mprotect, what's the cleanest way to do that, do I need to declare my own prototype for the function and its flags, or should I write some C code that does the call? Thanks for your help.
Jul 31 2012
On 31-07-2012 19:23, Maxime Chevalier wrote:New to the D language here. I'm working on a tracing JIT compiler and will need to allocate chunks of memory that are marked as executable. Is there a standard way of doing this in D, or do I need to directly call into mprotect? If I'm going to be calling mprotect, what's the cleanest way to do that, do I need to declare my own prototype for the function and its flags, or should I write some C code that does the call? Thanks for your help.
There's no standard way to do it. You'll have to either use mprotect or just pass the relevant flags when you call mmap. The functions and constants related to this are all in core.sys.posix.mman. See also: * https://github.com/lycus/mci/blob/master/src/mci/core/memory.d#L71 (and further down) * https://github.com/lycus/mci/blob/master/src/mci/vm/code.d -- Alex Rønne Petersen alex lycus.org http://lycus.org
Jul 31 2012
On 31-07-2012 19:36, Maxime Chevalier wrote:The functions and constants related to this are all in core.sys.posix.mman. See also: * https://github.com/lycus/mci/blob/master/src/mci/core/memory.d#L71 (and further down) * https://github.com/lycus/mci/blob/master/src/mci/vm/code.d
Thanks. Forgive my ignorance though, but does core.sys.posix.sys.mman ship standard with D compilers?
Yep. Anything core.* and std.* is part of druntime and phobos which both ship with any D 2.0 compiler. -- Alex Rønne Petersen alex lycus.org http://lycus.org
Jul 31 2012
=?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> On 31-07-2012 20:06, Maxime Chevalier wrote:Yep. Anything core.* and std.* is part of druntime and phobos which both ship with any D 2.0 compiler.
Thanks very much for the quick responses. My tracing JIT will likely need to have the compiled code write into data structures of the host VM. This will mean the compiled code holds pointers to host data. How well does the D GC deal with immovable objects. Would it be a problem if I allocated many chunks of immovable data?
Can you clarify what you mean by immovable data? I don't think I follow. In any case, if the compiled code's executable memory regions aren't added as root ranges to D's GC (which is entirely reasonable to not do), then you can keep the data alive by simply keeping pointers to the D GC-managed data in the host environment. D doesn't use a moving collector.Would it be preferable to minimize immovable data by having compiled code refer to an (immovable) table with pointers to (movable) host data? Also, if you don't mind me asking, which D compiler do you guys prefer? Which one is most widely supported, most up to date?
DMD: Latest updates, fixes, and enhancements. Very fast compilation. Not that great optimization. GDC: Best optimization and fantastic debug info generation. Somewhat slow at compilation. LDC: Fast compilation and good optimization. Sits somewhere between DMD and GDC. Note that DMD can only target x86 while LDC and GDC (due to using LLVM and GCC respectively) can target many other 32-bit and 64-bit architectures. Also, LDC and GDC tend to lag one release behind DMD sometimes, since they rely on the DMD front end code. -- Alex Rønne Petersen alex lycus.org http://lycus.org
Jul 31 2012
On 07/31/2012 08:06 PM, Maxime Chevalier wrote:How well does the D GC deal with immovable objects. Would it be a problem if I allocated many chunks of immovable data?
The current GC in druntime never moves data.
Jul 31 2012
The functions and constants related to this are all in core.sys.posix.mman. See also: * https://github.com/lycus/mci/blob/master/src/mci/core/memory.d#L71 (and further down) * https://github.com/lycus/mci/blob/master/src/mci/vm/code.d
Thanks. Forgive my ignorance though, but does core.sys.posix.sys.mman ship standard with D compilers?
Jul 31 2012
--bcaec554d63c0a889504c623be90 Content-Type: text/plain; charset=UTF-8 On Tue, Jul 31, 2012 at 9:36 PM, Maxime Chevalier < maximechevalierb gmail.com> wrote:The functions and constants related to this are all in core.sys.posix.mman.See also: * https://github.com/lycus/mci/**blob/master/src/mci/core/**memory.d#L71<https://github.com/lycus/mci/blob/master/src/mci/co e/memory.d#L71>(and further down) * https://github.com/lycus/mci/**blob/master/src/mci/vm/code.d<https://github.com/lycus/mci/blob/master/src/mci/vm/code.d>
Thanks. Forgive my ignorance though, but does core.sys.posix.sys.mman ship standard with D compilers?
It's part of druntime. Druntime is D's runtime library, without which D code above version 2.0 can't work. -- Bye, Gor Gyolchanyan. --bcaec554d63c0a889504c623be90 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div class=3D"gmail_quote">On Tue, Jul 31, 2012 at 9:36 PM, Maxime Chevalie= r <span dir=3D"ltr"><<a href=3D"mailto:maximechevalierb gmail.com" targe= t=3D"_blank">maximechevalierb gmail.com</a>></span> wrote:<br><blockquot= e class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc sol= id;padding-left:1ex"> <div class=3D"im"><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .= 8ex;border-left:1px #ccc solid;padding-left:1ex"> The functions and constants related to this are all in core.sys.posix.mman.= <br> <br> See also:<br> <br> * <a href=3D"https://github.com/lycus/mci/blob/master/src/mci/core/memory.d= #L71" target=3D"_blank">https://github.com/lycus/mci/<u></u>blob/master/src= /mci/core/<u></u>memory.d#L71</a> (and further down)<br> * <a href=3D"https://github.com/lycus/mci/blob/master/src/mci/vm/code.d" ta= rget=3D"_blank">https://github.com/lycus/mci/<u></u>blob/master/src/mci/vm/= code.d</a><br> </blockquote> <br></div> Thanks. Forgive my ignorance though, but does core.sys.posix.sys.mman ship = standard with D compilers?<br> </blockquote></div><br>It's part of druntime. Druntime is D's runti= me library, without which D code above version 2.0 can't work.<br clear= =3D"all"><div><br></div>-- <br>Bye,<br>Gor Gyolchanyan.<br> --bcaec554d63c0a889504c623be90--
Jul 31 2012
Yep. Anything core.* and std.* is part of druntime and phobos which both ship with any D 2.0 compiler.
Thanks very much for the quick responses. My tracing JIT will likely need to have the compiled code write into data structures of the host VM. This will mean the compiled code holds pointers to host data. How well does the D GC deal with immovable objects. Would it be a problem if I allocated many chunks of immovable data? Would it be preferable to minimize immovable data by having compiled code refer to an (immovable) table with pointers to (movable) host data? Also, if you don't mind me asking, which D compiler do you guys prefer? Which one is most widely supported, most up to date?
Jul 31 2012
On Tuesday, 31 July 2012 at 17:23:08 UTC, Maxime Chevalier wrote:New to the D language here. I'm working on a tracing JIT compiler and will need to allocate chunks of memory that are marked as executable. Is there a standard way of doing this in D, or do I need to directly call into mprotect? If I'm going to be calling mprotect, what's the cleanest way to do that, do I need to declare my own prototype for the function and its flags, or should I write some C code that does the call?
The x86 chip it's a simple flag that the OS can set. For use with like UPX, the whole section is marked read, write & execute I believe (since it has to expand it first before it can execute the code; That is of course for loaded memory, not allocating...). I would say check their sources, may prove interesting.
Jul 31 2012
On 31-07-2012 20:30, Era Scarecrow wrote:On Tuesday, 31 July 2012 at 17:23:08 UTC, Maxime Chevalier wrote:New to the D language here. I'm working on a tracing JIT compiler and will need to allocate chunks of memory that are marked as executable. Is there a standard way of doing this in D, or do I need to directly call into mprotect? If I'm going to be calling mprotect, what's the cleanest way to do that, do I need to declare my own prototype for the function and its flags, or should I write some C code that does the call?
The x86 chip it's a simple flag that the OS can set. For use with like UPX, the whole section is marked read, write & execute I believe (since it has to expand it first before it can execute the code; That is of course for loaded memory, not allocating...). I would say check their sources, may prove interesting.
UPX simply uses VirtualProtect/mprotect. It's the only way you _can_ do this in ring 3 on any OS with sane security. -- Alex Rønne Petersen alex lycus.org http://lycus.org
Jul 31 2012
Maxime Chevalier:How well does the D GC deal with immovable objects. Would it be a problem if I allocated many chunks of immovable data?
The GC supports flags to define memory as movable or immovable, but I think at the moment it performs no memory movements.Also, if you don't mind me asking, which D compiler do you guys prefer? Which one is most widely supported, most up to date?
DMD has a back-end that produces less efficient binaries compared to GDC and LDC. But DMD is more up to date (but not a lot), and it's the most supported, it's the reference implementation. DMD works well on Windows too, GDC works on Windows too. I think LDC doesn't work well on Windows. Bye, bearophile
Jul 31 2012
bearophile:DMD has a back-end that produces less efficient binaries compared to GDC and LDC. But DMD is more up to date (but not a lot), and it's the most supported, it's the reference implementation. DMD works well on Windows too, GDC works on Windows too. I think LDC doesn't work well on Windows.
An DMD doesn't support 64 bit on Windows. Bye, bearophile
Jul 31 2012
On Tuesday, 31 July 2012 at 18:06:28 UTC, Maxime Chevalier wrote:How well does the D GC deal with immovable objects. Would it be a problem if I allocated many chunks of immovable data?
D's GC currently doesn't move memory at all, so this should not be a problem. If there is a chance that all references to the object in question in D-managed memory (i.e. stack, GC heap) go away, but the memory is still used, you might need to manually mark the object as live, though (see core.memory.GC.addRoot).Also, if you don't mind me asking, which D compiler do you guys prefer? Which one is most widely supported, most up to date?
DMD is the official reference compiler and compiles faster than the others, but is x86-only (32 bit only on Windows), and the performance of the generated code is sometimes significantly worse than with the other compilers. GDC and LDC also use the same frontend as DMD, but it usually takes them a few weeks to catch up after a new version of it is released. LDC currently doesn't work on Windows if your application uses exception handling (which most applications do). I personally use LDC as my default compiler, but I might be biased… ;) David
Jul 31 2012
On Tuesday, 31 July 2012 at 17:23:08 UTC, Maxime Chevalier wrote:New to the D language here. I'm working on a tracing JIT compiler and will need to allocate chunks of memory that are marked as executable. Is there a standard way of doing this in D, or do I need to directly call into mprotect? If I'm going to be calling mprotect, what's the cleanest way to do that, do I need to declare my own prototype for the function and its flags, or should I write some C code that does the call? Thanks for your help.
Hope this helps: http://stackoverflow.com/a/8656294/21501
Jul 31 2012