www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - Identifier (...) is too long by X characters

reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
I just got such an error while trying to instantiate a rather longish 
template. This is pretty much a blocker for more complex template code. 
Couldn't symbol names be truncated at some predefined length ?

It also seems like long symbols are killing DMD performance-wise as they 
can eat lots of RAM while compiling. Then mem allocations + cache misses 
+ GC stalls make it crawl. I've tried instantiating a struct template 
(the Tuple from my 'bind' lib) with a nesting level of 50 and before I 
managed to kill DMD, it ate 700MB of virtual mem out of my 512MB RAM system.

Can anything be done with it ? It seems like compile times and memory 
use explode when dealing with more complex templates.


-- 
Tomasz Stachowiak  /+ a.k.a. h3r3tic +/
Jul 08 2006
parent reply John Reimer <John_member pathlink.com> writes:
In article <e8pf88$22ki$1 digitaldaemon.com>, Tom S says...
I just got such an error while trying to instantiate a rather longish 
template. This is pretty much a blocker for more complex template code. 
Couldn't symbol names be truncated at some predefined length ?

It also seems like long symbols are killing DMD performance-wise as they 
can eat lots of RAM while compiling. Then mem allocations + cache misses 
+ GC stalls make it crawl. I've tried instantiating a struct template 
(the Tuple from my 'bind' lib) with a nesting level of 50 and before I 
managed to kill DMD, it ate 700MB of virtual mem out of my 512MB RAM system.

Can anything be done with it ? It seems like compile times and memory 
use explode when dealing with more complex templates.


-- 
Tomasz Stachowiak  /+ a.k.a. h3r3tic +/

Tsk, tsk, tsk... what mad template experiments are you up to now, Tom? Your computer must be smoking at this point. lol! -JJR
Jul 09 2006
parent reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
John Reimer wrote:
 Tsk, tsk, tsk... what mad template experiments are you up to now, Tom?  Your
 computer must be smoking at this point. lol!

You know... the usual stuff ;D I'm just working with a *bit* more complex mixins, like this one: mixin Input!(`in`) .field!(DataStream!(vec3), `positions`) .field!(DataStream!(vec3), `normals`) .end input; When I add too many fields, DMD eats many many rams and gives me that error. -- Tomasz Stachowiak /+ a.k.a. h3r3tic +/
Jul 09 2006
parent reply Chris Nicholson-Sauls <ibisbasenji gmail.com> writes:
Tom S wrote:
 ...DMD eats many many rams...

*Gasp!* Those poor sheep/goats! I would be interested in seeing exactly what your 'Input' mixin does. The use of self-referancing returns to initialize it is pretty nifty though. Might be a way to get rid of the '.end' at the end, though? The error baffles me, though. -- Chris Nicholson-Sauls
Jul 09 2006
next sibling parent reply Sean Kelly <sean f4.ca> writes:
Chris Nicholson-Sauls wrote:
 Tom S wrote:
 ...DMD eats many many rams...

*Gasp!* Those poor sheep/goats! I would be interested in seeing exactly what your 'Input' mixin does. The use of self-referancing returns to initialize it is pretty nifty though. Might be a way to get rid of the '.end' at the end, though? The error baffles me, though.

Symbols are limited to ~255 chars on Win32. I think it's a limitation of the object file format. "symbol too long" errors are painfully common in C++, though they're typically truncated to the max length to allow for debugging. Sean
Jul 09 2006
parent reply Don Clugston <dac nospam.com.au> writes:
Sean Kelly wrote:
 Chris Nicholson-Sauls wrote:
 Tom S wrote:
 ...DMD eats many many rams...

*Gasp!* Those poor sheep/goats! I would be interested in seeing exactly what your 'Input' mixin does. The use of self-referancing returns to initialize it is pretty nifty though. Might be a way to get rid of the '.end' at the end, though? The error baffles me, though.

Symbols are limited to ~255 chars on Win32. I think it's a limitation of the object file format. "symbol too long" errors are painfully common in C++, though they're typically truncated to the max length to allow for debugging.

I thought the OMF lib-imposed limit was about 4K. I've just made a template with a symbol name of 480 characters, and that was OK. If Tuple!() works with a nesting level of 50, the limit would have to be pretty big. Maybe in the future, we'll be able to add some kind of 'final' declaration to templates, which would indicate that they don't need to be stored in the object file or library, and can be discarded from the symbol table as soon as they are instantiated. This would be particularly useful for recursive templates, which can easily spew reams of garbage into the object file.
Jul 09 2006
next sibling parent reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Don Clugston wrote:
 If Tuple!() works with a nesting level of 50, the limit would have to be 
 pretty big.

Yeah, and it would cause DMD to eat RAMs like a medium-sized dragon.
 Maybe in the future, we'll be able to add some kind of 'final' 
 declaration to templates, which would indicate that they don't need to 
 be stored in the object file or library, and can be discarded from the 
 symbol table as soon as they are instantiated. This would be 
 particularly useful for recursive templates, which can easily spew reams 
 of garbage into the object file.

My thoughts exactly :) Right now I'd be happy with anything that worked though... And while we're at recursive templates, don't you think that the limitation for recursive templates, which causes the error 'template instance recursive template expansion for template argument X' is pretty arbitary ? It can be hacked around, after all: // tmp.d import std.stdio; struct Foo(T/*, int dummy=0*/) { Foo!(Foo!(T/*, dummy+1*/)) rec()() { // line 5 Foo!(Foo!(T/*, dummy+1*/)) res; return res; } } void main() { Foo!(int) a; writefln(typeid(typeof(a))); auto b = a.rec!()(); writefln(typeid(typeof(b))); } // ---- tmp.d(5): template instance recursive template expansion for template argument Foo But when you uncomment the 'dummies', it compiles and runs without any problems... Yeah, I know, the error is an artifact of how the instantiation works, that it's not 'lazy' like in C++. But since it can be fooled into working, couldn't DMD apply the hack automagically ? -- Tomasz Stachowiak /+ a.k.a. h3r3tic +/
Jul 10 2006
parent reply Sean Kelly <sean f4.ca> writes:
Tom S wrote:
 Don Clugston wrote:
 If Tuple!() works with a nesting level of 50, the limit would have to 
 be pretty big.

Yeah, and it would cause DMD to eat RAMs like a medium-sized dragon.
 Maybe in the future, we'll be able to add some kind of 'final' 
 declaration to templates, which would indicate that they don't need to 
 be stored in the object file or library, and can be discarded from the 
 symbol table as soon as they are instantiated. This would be 
 particularly useful for recursive templates, which can easily spew 
 reams of garbage into the object file.

My thoughts exactly :) Right now I'd be happy with anything that worked though... And while we're at recursive templates, don't you think that the limitation for recursive templates, which causes the error 'template instance recursive template expansion for template argument X' is pretty arbitary ? It can be hacked around, after all: // tmp.d import std.stdio; struct Foo(T/*, int dummy=0*/) { Foo!(Foo!(T/*, dummy+1*/)) rec()() { // line 5 Foo!(Foo!(T/*, dummy+1*/)) res; return res; } } void main() { Foo!(int) a; writefln(typeid(typeof(a))); auto b = a.rec!()(); writefln(typeid(typeof(b))); } // ---- tmp.d(5): template instance recursive template expansion for template argument Foo But when you uncomment the 'dummies', it compiles and runs without any problems... Yeah, I know, the error is an artifact of how the instantiation works, that it's not 'lazy' like in C++. But since it can be fooled into working, couldn't DMD apply the hack automagically ?

I don't know. To do so would be like allowing: struct S { S val; } I'm not sure I'd want the compiler to try and "fix" a mistake I'd made involving this. Adding an integer is pretty easy anyway. Sean
Jul 10 2006
parent reply Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Sean Kelly wrote:
 I don't know.  To do so would be like allowing:
 
 struct S
 {
     S val;
 }

Umm... not quite. In your case it's just a plain nested, recursive struct which is obviously impossible. In my case: struct Foo(T) { Foo!(Foo!(T)) rec()() { Foo!(Foo!(T)) res; return res; } } When the function template is instantiated, one more nesting level is 'added' and a non-recurring struct is returned. The compiler reports an error because somehow it doesn't instantiate the 'rec' template lazily. In the example you've given there's an infinite theoretical nesting, yet in the version with the function, infinite nesting isn't possible, because the nesting increases one level with each 'rec' instantiation. IIRC, that's perfectly legit in C++ as it uses a different method for template instantiation.
 I'm not sure I'd want the compiler to try and "fix" a mistake I'd made 
 involving this.  Adding an integer is pretty easy anyway.

Easy ? It's undocumented, tedious and generally weird. It might stop working in the next compiler version because I'm not even sure why DMD allows it. -- Tomasz Stachowiak /+ a.k.a. h3r3tic +/
Jul 10 2006
next sibling parent reply Sean Kelly <sean f4.ca> writes:
Tom S wrote:
 Sean Kelly wrote:
 I don't know.  To do so would be like allowing:

 struct S
 {
     S val;
 }

Umm... not quite. In your case it's just a plain nested, recursive struct which is obviously impossible. In my case: struct Foo(T) { Foo!(Foo!(T)) rec()() { Foo!(Foo!(T)) res; return res; } } When the function template is instantiated, one more nesting level is 'added' and a non-recurring struct is returned.

Oops, you're right. I should have looked at the example more closely.
 The compiler reports an
 error because somehow it doesn't instantiate the 'rec' template lazily. 
 In the example you've given there's an infinite theoretical nesting, yet 
 in the version with the function, infinite nesting isn't possible, 
 because the nesting increases one level with each 'rec' instantiation. 
 IIRC, that's perfectly legit in C++ as it uses a different method for 
 template instantiation.

Yeah. I think the problem in D is that all template members are explicitly instantiated together instead of on an as-needed basis. But as rec() is itself a template function I wouldn't have expected this behavior. I agree that this should be changed, assuming it's possible to do so without entirely changing the way templates are instantiated in D.
 I'm not sure I'd want the compiler to try and "fix" a mistake I'd made 
 involving this.  Adding an integer is pretty easy anyway.

Easy ? It's undocumented, tedious and generally weird. It might stop working in the next compiler version because I'm not even sure why DMD allows it.

You're right. Sorry for the confusion. Sean
Jul 10 2006
parent Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Sean Kelly wrote:
 Tom S wrote:
 Sean Kelly wrote:
 I don't know.  To do so would be like allowing:

 struct S
 {
     S val;
 }

Umm... not quite. In your case it's just a plain nested, recursive struct which is obviously impossible. In my case: struct Foo(T) { Foo!(Foo!(T)) rec()() { Foo!(Foo!(T)) res; return res; } } When the function template is instantiated, one more nesting level is 'added' and a non-recurring struct is returned.

Oops, you're right. I should have looked at the example more closely.

Ah, you haven't noticed the second pair of parentheses after 'rec' ?
 Yeah.  I think the problem in D is that all template members are 
 explicitly instantiated together instead of on an as-needed basis.  But 
 as rec() is itself a template function I wouldn't have expected this 
 behavior.  I agree that this should be changed, assuming it's possible 
 to do so without entirely changing the way templates are instantiated in D.

Thanks for support :) -- Tomasz Stachowiak /+ a.k.a. h3r3tic +/
Jul 10 2006
prev sibling parent Bruno Medeiros <brunodomedeirosATgmail SPAM.com> writes:
Tom S wrote:
 Sean Kelly wrote:
 I don't know.  To do so would be like allowing:

 struct S
 {
     S val;
 }

Umm... not quite. In your case it's just a plain nested, recursive struct which is obviously impossible. In my case: struct Foo(T) { Foo!(Foo!(T)) rec()() { Foo!(Foo!(T)) res; return res; } } When the function template is instantiated, one more nesting level is 'added' and a non-recurring struct is returned. The compiler reports an error because somehow it doesn't instantiate the 'rec' template lazily. In the example you've given there's an infinite theoretical nesting, yet in the version with the function, infinite nesting isn't possible, because the nesting increases one level with each 'rec' instantiation. IIRC, that's perfectly legit in C++ as it uses a different method for template instantiation.
 I'm not sure I'd want the compiler to try and "fix" a mistake I'd made 
 involving this.  Adding an integer is pretty easy anyway.

Easy ? It's undocumented, tedious and generally weird. It might stop working in the next compiler version because I'm not even sure why DMD allows it.

The compiler does always evaluate "lazily", and that is documented, although it may not be immediately obvious: The template spec states that "Semantic analysis is not done until instantiated." which in other words means a template is not evaluated until instantiated. This remains true for nested templates, and thus for your example as well. Then why the error message? My guess is that the error message is simply a mistake on part of the compiler, where the recursive template detection system gives a false positive (it thinks it will expand indefinitely but it won't). Thus it is a bug. -- Bruno Medeiros - CS/E student http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Jul 12 2006
prev sibling next sibling parent reply Sean Kelly <sean f4.ca> writes:
Don Clugston wrote:
 Sean Kelly wrote:
 Chris Nicholson-Sauls wrote:
 Tom S wrote:
 ...DMD eats many many rams...

*Gasp!* Those poor sheep/goats! I would be interested in seeing exactly what your 'Input' mixin does. The use of self-referancing returns to initialize it is pretty nifty though. Might be a way to get rid of the '.end' at the end, though? The error baffles me, though.

Symbols are limited to ~255 chars on Win32. I think it's a limitation of the object file format. "symbol too long" errors are painfully common in C++, though they're typically truncated to the max length to allow for debugging.

I thought the OMF lib-imposed limit was about 4K. I've just made a template with a symbol name of 480 characters, and that was OK.

Perhaps it is--it's been a while since I developed on Windows. I only remember seeing the errors quite often.
 Maybe in the future, we'll be able to add some kind of 'final' 
 declaration to templates, which would indicate that they don't need to 
 be stored in the object file or library, and can be discarded from the 
 symbol table as soon as they are instantiated. This would be 
 particularly useful for recursive templates, which can easily spew reams 
 of garbage into the object file.

This would be nice. I generally expect this to occur anyway, but it would be nice to have some sort of insurance. Sean
Jul 10 2006
parent Don Clugston <dac nospam.com.au> writes:
Sean Kelly wrote:
 Don Clugston wrote:
 Sean Kelly wrote:
 Chris Nicholson-Sauls wrote:
 Tom S wrote:
 ...DMD eats many many rams...

*Gasp!* Those poor sheep/goats! I would be interested in seeing exactly what your 'Input' mixin does. The use of self-referancing returns to initialize it is pretty nifty though. Might be a way to get rid of the '.end' at the end, though? The error baffles me, though.

Symbols are limited to ~255 chars on Win32. I think it's a limitation of the object file format. "symbol too long" errors are painfully common in C++, though they're typically truncated to the max length to allow for debugging.

I thought the OMF lib-imposed limit was about 4K. I've just made a template with a symbol name of 480 characters, and that was OK.

Perhaps it is--it's been a while since I developed on Windows. I only remember seeing the errors quite often.
 Maybe in the future, we'll be able to add some kind of 'final' 
 declaration to templates, which would indicate that they don't need to 
 be stored in the object file or library, and can be discarded from the 
 symbol table as soon as they are instantiated. This would be 
 particularly useful for recursive templates, which can easily spew 
 reams of garbage into the object file.

This would be nice. I generally expect this to occur anyway, but it would be nice to have some sort of insurance.

It definitely happens, but I don't understand why they're in the obj at all (if I understand correctly, it has to happen that way in C++ because it's possible for a seperate obj file to provide a specialisation; but that's not an issue for D). It's a real problem for the use of metaprogramming with DDL. The .objs can a be a megabyte in size, but contribute only a hundred bytes to the final exe. It would make linking much faster if it didn't happen, and it would probably improve compilation time.
Jul 10 2006
prev sibling parent reply Bruno Medeiros <brunodomedeirosATgmail SPAM.com> writes:
Don Clugston wrote:
 Sean Kelly wrote:
 Chris Nicholson-Sauls wrote:
 Tom S wrote:
 ...DMD eats many many rams...

*Gasp!* Those poor sheep/goats! I would be interested in seeing exactly what your 'Input' mixin does. The use of self-referancing returns to initialize it is pretty nifty though. Might be a way to get rid of the '.end' at the end, though? The error baffles me, though.

Symbols are limited to ~255 chars on Win32. I think it's a limitation of the object file format. "symbol too long" errors are painfully common in C++, though they're typically truncated to the max length to allow for debugging.

I thought the OMF lib-imposed limit was about 4K. I've just made a template with a symbol name of 480 characters, and that was OK. If Tuple!() works with a nesting level of 50, the limit would have to be pretty big. Maybe in the future, we'll be able to add some kind of 'final' declaration to templates, which would indicate that they don't need to be stored in the object file or library, and can be discarded from the symbol table as soon as they are instantiated. This would be particularly useful for recursive templates, which can easily spew reams of garbage into the object file.

What are these templates that can be discarded from the object file? Is it templates that only contain compile-time members? Do these even take space in an object file (since they have no run-time members) ? -- Bruno Medeiros - CS/E student http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Jul 14 2006
parent reply Don Clugston <dac nospam.com.au> writes:
Bruno Medeiros wrote:
 Don Clugston wrote:
 Sean Kelly wrote:
 Chris Nicholson-Sauls wrote:
 Tom S wrote:
 ...DMD eats many many rams...

*Gasp!* Those poor sheep/goats! I would be interested in seeing exactly what your 'Input' mixin does. The use of self-referancing returns to initialize it is pretty nifty though. Might be a way to get rid of the '.end' at the end, though? The error baffles me, though.

Symbols are limited to ~255 chars on Win32. I think it's a limitation of the object file format. "symbol too long" errors are painfully common in C++, though they're typically truncated to the max length to allow for debugging.

I thought the OMF lib-imposed limit was about 4K. I've just made a template with a symbol name of 480 characters, and that was OK. If Tuple!() works with a nesting level of 50, the limit would have to be pretty big. Maybe in the future, we'll be able to add some kind of 'final' declaration to templates, which would indicate that they don't need to be stored in the object file or library, and can be discarded from the symbol table as soon as they are instantiated. This would be particularly useful for recursive templates, which can easily spew reams of garbage into the object file.

What are these templates that can be discarded from the object file? Is it templates that only contain compile-time members? Do these even take space in an object file (since they have no run-time members) ?

Yes, they take up a lot of space in the object file (can be megabytes). eg, if the source file contains int f = factorial!(7), then the object file will also contain the useless intermediates factorial!(6), factorial!(5),...factorial!(1). The unused ones don't get discarded until link time. I'm not sure why they're in the obj file at all, but they certainly get in there.
Jul 14 2006
parent Bruno Medeiros <brunodomedeirosATgmail SPAM.com> writes:
Don Clugston wrote:
 Bruno Medeiros wrote:
 Don Clugston wrote:
 Sean Kelly wrote:
 Chris Nicholson-Sauls wrote:
 Tom S wrote:
 ...DMD eats many many rams...

*Gasp!* Those poor sheep/goats! I would be interested in seeing exactly what your 'Input' mixin does. The use of self-referancing returns to initialize it is pretty nifty though. Might be a way to get rid of the '.end' at the end, though? The error baffles me, though.

Symbols are limited to ~255 chars on Win32. I think it's a limitation of the object file format. "symbol too long" errors are painfully common in C++, though they're typically truncated to the max length to allow for debugging.

I thought the OMF lib-imposed limit was about 4K. I've just made a template with a symbol name of 480 characters, and that was OK. If Tuple!() works with a nesting level of 50, the limit would have to be pretty big. Maybe in the future, we'll be able to add some kind of 'final' declaration to templates, which would indicate that they don't need to be stored in the object file or library, and can be discarded from the symbol table as soon as they are instantiated. This would be particularly useful for recursive templates, which can easily spew reams of garbage into the object file.

What are these templates that can be discarded from the object file? Is it templates that only contain compile-time members? Do these even take space in an object file (since they have no run-time members) ?

Yes, they take up a lot of space in the object file (can be megabytes). eg, if the source file contains int f = factorial!(7), then the object file will also contain the useless intermediates factorial!(6), factorial!(5),...factorial!(1). The unused ones don't get discarded until link time. I'm not sure why they're in the obj file at all, but they certainly get in there.

Then that means a language construct would not be necessary, as the compiler or linker should be able themselves to know not to generate empty template sections. And that seems easy to implement, at least at first sight (if it requires changing optlink we might not be so lucky). -- Bruno Medeiros - CS/E student http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Jul 15 2006
prev sibling parent Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
Chris Nicholson-Sauls wrote:
 Tom S wrote:
 ...DMD eats many many rams...

*Gasp!* Those poor sheep/goats! I would be interested in seeing exactly what your 'Input' mixin does.

There you go: http://158.75.59.9/~h3/code/processing.rar It's in processing/input.d
 The use of self-referancing returns to initialize it is pretty nifty 
 though.  Might be a way to get rid of the '.end' at the end, though?

Hmm... check the code and decide for yourself... I thought using the 'end' was the most straightforward approach. Otherwise the implicit name promotion wouldn't work and I'd have to use field.field all over the place. 'field.add' could work, but I considered forcing the user to add one 'end' to be simpler. -- Tomasz Stachowiak /+ a.k.a. h3r3tic +/
Jul 10 2006