www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Heap memory limit?

reply bearophile <bearophileHUGS lycos.com> writes:
The Java VM has a limit of the amount of memory it allocates from the heap. So
is it useful to add a way to specify a similar limit for D2 programs
(especially ones made by safe module)? This limit can be given to the D GC heap
(and maybe to the C heap too).

This limit can just be a compile-time constant given to the compiler, this
produces a binary that has a limit of 200 MB of heap:
dmd -mlimit=200mb foo.d
The situation gets less simple if you want to change such limit after the
program is already compiled, I don't know of simple ways to do it. Maybe a
second program can be used for that:
gclimiter -mlimit=100mb foo.exe

Bye,
bearophile
Feb 23 2010
next sibling parent reply "Lars T. Kyllingstad" <public kyllingen.NOSPAMnet> writes:
bearophile wrote:
 The Java VM has a limit of the amount of memory it allocates from the heap. So
is it useful to add a way to specify a similar limit for D2 programs
(especially ones made by safe module)? This limit can be given to the D GC heap
(and maybe to the C heap too).
 
 This limit can just be a compile-time constant given to the compiler, this
produces a binary that has a limit of 200 MB of heap:
 dmd -mlimit=200mb foo.d
 The situation gets less simple if you want to change such limit after the
program is already compiled, I don't know of simple ways to do it. Maybe a
second program can be used for that:
 gclimiter -mlimit=100mb foo.exe

On Linux you can use the shell's built-in 'ulimit' command to set such a limit: $ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 20 file size (blocks, -f) unlimited pending signals (-i) 16382 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited -Lars
Feb 23 2010
parent bearophile <bearophileHUGS lycos.com> writes:
Lars T. Kyllingstad:
 On Linux you can use the shell's built-in 'ulimit' command to set such a 
 limit

My idea was the opposite (and similar to how Java works on Windows): to set a default limit that's always present unless you specify you want more memory. As usual, if using too much memory is seen as a danger (and it can make a system unstable, so I think it's a small safety thing, that's probably why Java has such limit), then to avoid such danger you have make the default behaviour the safer one. Bye, bearophile
Feb 23 2010
prev sibling next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 23 Feb 2010 08:36:04 -0500, bearophile <bearophileHUGS lycos.com>  
wrote:

 The Java VM has a limit of the amount of memory it allocates from the  
 heap. So is it useful to add a way to specify a similar limit for D2  
 programs (especially ones made by safe module)? This limit can be given  
 to the D GC heap (and maybe to the C heap too).

 This limit can just be a compile-time constant given to the compiler,  
 this produces a binary that has a limit of 200 MB of heap:
 dmd -mlimit=200mb foo.d
 The situation gets less simple if you want to change such limit after  
 the program is already compiled, I don't know of simple ways to do it.  
 Maybe a second program can be used for that:
 gclimiter -mlimit=100mb foo.exe

The GC is open source. Submit a patch if you want. DMD is not in charge of the GC, it is runtime code, I don't think we need a specific option for it. I think it can be runtime-decided, and the option passed via your program initialization. I.e.: void main() { GC.memlimit = 200_000_000; // 200 MB ... } I also don't think that limits to memory are too important for most applications, only for specific ones. -Steve
Feb 23 2010
parent reply bearophile <bearophileHUGS lycos.com> writes:
Steven Schveighoffer:

 The GC is open source.  Submit a patch if you want.

It can be nice if the the C heap too is counted in such max memory used. I don't know if this is possible.
DMD is not in charge of the GC, it is runtime code, I don't think we need a
specific option for it.<

- D2 code has safe modules too, they are meant to turn D code into something a little safer than usual C code. - I think using too much memory is a unsafe situation, a small risk. You are free to not agree. - If it's seen as a real small risk, then if you want to do something concrete to avoid this risk, you need such memory limit enforced by default (if it's not a default, very few people will use it, and this small risk will be often ignored). And it can be positive if all D implementations share have this safety. That's why I was talking about a compiler argument (plus eventually a small program to change at runtime the limit of an already compiled program. Another silly possible solution is to use small program to modify the max value contained into the binary itself, a kind of binary patching). - If you want the original behaviour you can compile with something like: dmd -mlimit=none foo.d This disables such limiting (it disables the run-time tests too, so the allocations become a bit faster too).
 I also don't think that limits to memory are too important for most  
 applications, only for specific ones.

It's not just my desire to copy Java, I have made my PC unstable few times because a bug in my code has caused a D proggy to eat too much memory, so having a default limit of memory used seems useful to me. Bye, bearophile
Feb 23 2010
parent bearophile <bearophileHUGS lycos.com> writes:
Steven Schveighoffer:

Also, however, C's malloc isn't as likely to explode memory usage because items
should be freed when they are no longer used.<

Of course , if we assume perfect programmers, then there's no need of many other kinds of safeties present in D code.
 Again, this can be done in the GC code itself, it does not need to be a  
 compiler-defined limit.  My preference is for the default to be unlimited,  
 since this is the most common case.

This is wrong; in the most common case you don't want a program to trash your virtual memory and crash your other programs. Only for special programs you want no limits. I am now slowly starting to understand why new languages (Java, C# and all the ones created on dotnet and JVM) are designed the other way around compared to D: they start from a safe situation and then add (usually safe) optimizations to try to become faster. Because starting from C you have no hope of improving certain things (some of such things are limits in the brain of people used to C). Thank you for your answers and explanations, I can see there's no use of the limit I was talking about. Better to put an external limit to the program, as Lars Kyllingstad has shown, or maybe to add a memory limit to the GC, that can defined at run-time. -------------- Justin Johansson:
Can you please explain why you think this is useful?<

To make D programs a bit safer. On Windows if you have 5 programs running, and one of them eats too much RAM, the system starts swapping, everything starts running very slowly, some other programs crash, etc. This is not nice. So if I write a small buggy program to process a large amount of genomic data I'd like a bug in my code to avoid that out of memory situation. But Steven Schveighoffer has just explained why my idea can't work in D. So sorry for all the noise :-)
Assuming the idea is useful, why would you have to be a compiler option rather
than, say, a linker option?<

I don't care of linkers. They are an implementation detail that in a modern language I want to be able to ignore in most times, unless I'm doing something special (and the situation I am discussing here is not special at all, I think of it as the default one). To use a car designed in 2008 I don't want to do all the things you have to do to start a car designed in 1940 :-) I want it to work in a simpler way, even if inside it's probably more complex. Bye, bearophile
Feb 23 2010
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 23 Feb 2010 09:34:19 -0500, bearophile <bearophileHUGS lycos.com>  
wrote:

 Steven Schveighoffer:

 The GC is open source.  Submit a patch if you want.

It can be nice if the the C heap too is counted in such max memory used. I don't know if this is possible.

I don't know if this is possible. D would have to replace C's malloc in order to do this, but I don't think that's a good idea. Also, however, C's malloc isn't as likely to explode memory usage because items should be freed when they are no longer used. GC-based languages tend to use more memory because there are at any one time a lot of unused pieces of memory waiting to be cleaned.
 DMD is not in charge of the GC, it is runtime code, I don't think we  
 need a specific option for it.<

- D2 code has safe modules too, they are meant to turn D code into something a little safer than usual C code. - I think using too much memory is a unsafe situation, a small risk. You are free to not agree.

It is a risk, but it is entirely dependent on factors outside the program's control. For example, how much memory is available. If you limit to 200MB, what if you have to run 50 programs that were built from D with the same default limit, that's 10GB limit. I just don't think it's realistic to come up with a simple per-program limit that solves the problem.
 - If it's seen as a real small risk, then if you want to do something  
 concrete to avoid this risk, you need such memory limit enforced by  
 default (if it's not a default, very few people will use it, and this  
 small risk will be often ignored). And it can be positive if all D  
 implementations share have this safety. That's why I was talking about a  
 compiler argument (plus eventually a small program to change at runtime  
 the limit of an already compiled program. Another silly possible  
 solution is to use  small program to modify the max value contained into  
 the binary itself, a kind of binary patching).

Again, this can be done in the GC code itself, it does not need to be a compiler-defined limit. My preference is for the default to be unlimited, since this is the most common case.
 I also don't think that limits to memory are too important for most
 applications, only for specific ones.

It's not just my desire to copy Java, I have made my PC unstable few times because a bug in my code has caused a D proggy to eat too much memory, so having a default limit of memory used seems useful to me.

This isn't a common case though. An artificial limit that has no consideration of available memory seems like it would lead to more puzzlement than help. -Steve
Feb 23 2010
prev sibling next sibling parent Justin Johansson <no spam.com> writes:
Hi bearophile,

bearophile wrote:
 The Java VM has a limit of the amount of memory it allocates from the heap. So
is it useful to add a way to specify a similar limit for D2 programs
(especially ones made by safe module)? This limit can be given to the D GC heap
(and maybe to the C heap too).

Can you please explain why you think this is useful?
 This limit can just be a compile-time constant given to the compiler, this
produces a binary that has a limit of 200 MB of heap:
 dmd -mlimit=200mb foo.d
 The situation gets less simple if you want to change such limit after the
program is already compiled, I don't know of simple ways to do it. Maybe a
second program can be used for that:
 gclimiter -mlimit=100mb foo.exe

Assuming the idea is useful, why would you have to be a compiler option rather than, say, a linker option? Regards, Justin
Feb 23 2010
prev sibling parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 23 Feb 2010 08:36:04 -0500, bearophile <bearophileHUGS lycos.com>  
wrote:
 The Java VM has a limit of the amount of memory it allocates from the  
 heap. So is it useful to add a way to specify a similar limit for D2  
 programs (especially ones made by safe module)? This limit can be given  
 to the D GC heap (and maybe to the C heap too).

 This limit can just be a compile-time constant given to the compiler,  
 this produces a binary that has a limit of 200 MB of heap:
 dmd -mlimit=200mb foo.d
 The situation gets less simple if you want to change such limit after  
 the program is already compiled, I don't know of simple ways to do it.  
 Maybe a second program can be used for that:
 gclimiter -mlimit=100mb foo.exe

 Bye,
 bearophile

My gut feeling is that you are confusing free heap memory with total heap memory, because I know of lots of Java programs where I seamlessly allocate more than 200mb on a regular basis. I mean, consider a user with gigs of memory free suddenly getting an out of memory error. Anyways, as others have mentioned, this seems like a fairly trivial addition to druntime's GC.
Feb 23 2010
parent bearophile <bearophileHUGS lycos.com> writes:
Robert Jacques:

Anyways, as others have mentioned, this seems like a fairly trivial addition to
druntime's GC.<

OK, and thank you. Bye, bearophile
Feb 23 2010