www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Huge pages and druntime parameters

reply "Martin Nowak" <code dawg.eu> writes:
Running dmd with huge page backed malloc resulted in about 10% 
faster compilations.
This looks very promising though other tests with some D GC 
microbenchmarks did not benefit from huge pages.
You can read more about this here.
http://www.ibm.com/developerworks/systems/library/es-lop-leveragepages/

I'm interested in adding a runtime parameter that would advise 
the GC to preferably use huge pages as backing memory.
There are a number of other interesting runtime parameters, like 
max heap size, GC grow policies, turning on statistics, so I 
wonders how to make them available.
Java seems to use -XX:+UseLargePages and -Xmx for the maximum 
heap size.
GHC uses a nice scheme where runtime parameters passed via 
command line are embraced by +RTS arg1 arg2 -RTS. They also 
require to link-in support for this.
http://www.haskell.org/ghc/docs/7.0.1/html/users_guide/runtime-control.html
Any good ideas for druntime?
Sep 26 2013
next sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, September 27, 2013 02:31:21 Martin Nowak wrote:
 I'm interested in adding a runtime parameter that would advise
 the GC to preferably use huge pages as backing memory.

 Any good ideas for druntime?

IIRC, Leandro made his concurrent GC use environment variables to set stuff like this. - Jonathan M Davis
Sep 26 2013
parent Martin Nowak <code dawg.eu> writes:
On 09/27/2013 02:54 AM, Jonathan M Davis wrote:
 IIRC, Leandro made his concurrent GC use environment variables to set stuff
 like this.

 - Jonathan M Davis

COW because the virtual memory granularity goes from 4Kb to 2Mb, i.e. 500x more copying when the parent process modifies a page while the forked process performs GC.
Oct 01 2013
prev sibling next sibling parent reply "luminousone" <rd.hunt gmail.com> writes:
On Friday, 27 September 2013 at 00:31:23 UTC, Martin Nowak wrote:
 Running dmd with huge page backed malloc resulted in about 10% 
 faster compilations.
 This looks very promising though other tests with some D GC 
 microbenchmarks did not benefit from huge pages.
 You can read more about this here.
 http://www.ibm.com/developerworks/systems/library/es-lop-leveragepages/

 I'm interested in adding a runtime parameter that would advise 
 the GC to preferably use huge pages as backing memory.
 There are a number of other interesting runtime parameters, 
 like max heap size, GC grow policies, turning on statistics, so 
 I wonders how to make them available.
 Java seems to use -XX:+UseLargePages and -Xmx for the maximum 
 heap size.
 GHC uses a nice scheme where runtime parameters passed via 
 command line are embraced by +RTS arg1 arg2 -RTS. They also 
 require to link-in support for this.
 http://www.haskell.org/ghc/docs/7.0.1/html/users_guide/runtime-control.html
 Any good ideas for druntime?

I have played around with huge pages a bit, my understanding from reading about them, and use(ROAM diamond tree nodes in huge page memory), is that they need to be used carefully, many processors only have a small number of entries for huge pages in their lookup tables. The best use of huge pages seems to be from code built around them, not sure if it entirely makes sense to put them in gc,
Sep 26 2013
parent Martin Nowak <code dawg.eu> writes:
On 09/27/2013 04:01 AM, luminousone wrote:
 I have played around with huge pages a bit, my understanding from
 reading about them, and use(ROAM diamond tree nodes in huge page
 memory), is that they need to be used carefully, many processors only
 have a small number of entries for huge pages in their lookup tables.

When out of huge pages it would fallback to normal mmap.
 The best use of huge pages seems to be from code built around them, not
 sure if it entirely makes sense to put them in gc,

The main benefit should come from heavily reducing the amount of TLB entries in the cache. So if your program suffers from many cache misses due to TLB lookups that would help. And the freed cache can also be used for other stuff instead.
Oct 01 2013
prev sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 27.09.2013 02:31, Martin Nowak wrote:
 Running dmd with huge page backed malloc resulted in about 10% faster
 compilations.
 This looks very promising though other tests with some D GC
 microbenchmarks did not benefit from huge pages.
 You can read more about this here.
 http://www.ibm.com/developerworks/systems/library/es-lop-leveragepages/

 I'm interested in adding a runtime parameter that would advise the GC to
 preferably use huge pages as backing memory.
 There are a number of other interesting runtime parameters, like max
 heap size, GC grow policies, turning on statistics, so I wonders how to
 make them available.

An application that needs to allocate ore than a few MB is currently often slowed down considerably by early collections of the GC. Being able to use larger values for initial memory and pool size increments would help a lot here.
 Java seems to use -XX:+UseLargePages and -Xmx for the maximum heap size.
 GHC uses a nice scheme where runtime parameters passed via command line
 are embraced by +RTS arg1 arg2 -RTS. They also require to link-in
 support for this.
 http://www.haskell.org/ghc/docs/7.0.1/html/users_guide/runtime-control.html
 Any good ideas for druntime?

For the precise GC I moved the configuration variable(s) to its own module: https://github.com/rainers/druntime/blob/gcx_precise/src/gc/config.d This module allows configuration via environment variables, but you can change the defaults by adding a file with module declaration "gc.config" and other settings.
Sep 27 2013
parent Martin Nowak <code dawg.eu> writes:
On 09/27/2013 03:33 PM, Rainer Schuetze wrote:
 An application that needs to allocate ore than a few MB is currently
 often slowed down considerably by early collections of the GC. Being
 able to use larger values for initial memory and pool size increments
 would help a lot here.

 For the precise GC I moved the configuration variable(s) to its own
 module:
 https://github.com/rainers/druntime/blob/gcx_precise/src/gc/config.d
 This module allows configuration via environment variables, but you can
 change the defaults by adding a file with module declaration "gc.config"
 and other settings.

Oct 01 2013