www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - are mixin string templates with functions removed?

reply "JS" <js.mdnq gmail.com> writes:
If I have a bunch of templates that are only used for code 
generation, are they removed in the binary(since they are not 
needed)?
Aug 01 2013
parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Thursday, 1 August 2013 at 17:09:07 UTC, JS wrote:
 If I have a bunch of templates that are only used for code 
 generation, are they removed in the binary(since they are not 
 needed)?
templates don't exist in binaries.
Aug 01 2013
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Aug 01, 2013 at 07:12:51PM +0200, John Colvin wrote:
 On Thursday, 1 August 2013 at 17:09:07 UTC, JS wrote:
If I have a bunch of templates that are only used for code
generation, are they removed in the binary(since they are not
needed)?
templates don't exist in binaries.
Templates are like cookie molds, you use them to generate lots of (almost) identical cookies, but you never serve the mold to the customer. ;-) T -- He who does not appreciate the beauty of language is not worthy to bemoan its flaws.
Aug 01 2013
next sibling parent reply "JS" <js.mdnq gmail.com> writes:
On Thursday, 1 August 2013 at 17:47:00 UTC, H. S. Teoh wrote:
 On Thu, Aug 01, 2013 at 07:12:51PM +0200, John Colvin wrote:
 On Thursday, 1 August 2013 at 17:09:07 UTC, JS wrote:
If I have a bunch of templates that are only used for code
generation, are they removed in the binary(since they are not
needed)?
templates don't exist in binaries.
Templates are like cookie molds, you use them to generate lots of (almost) identical cookies, but you never serve the mold to the customer. ;-) T
But what about the functions that exist in them? e.g., template A() { void A() { B(); } void B() { } } is everything in the template removed 100% or is there junk that the compiler doesn't remove? Oh... and I'm not talking about theoretical... I'm talking about what dmd actually does.
Aug 01 2013
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Aug 01, 2013 at 07:52:28PM +0200, JS wrote:
 On Thursday, 1 August 2013 at 17:47:00 UTC, H. S. Teoh wrote:
On Thu, Aug 01, 2013 at 07:12:51PM +0200, John Colvin wrote:
On Thursday, 1 August 2013 at 17:09:07 UTC, JS wrote:
If I have a bunch of templates that are only used for code
generation, are they removed in the binary(since they are not
needed)?
templates don't exist in binaries.
Templates are like cookie molds, you use them to generate lots of (almost) identical cookies, but you never serve the mold to the customer. ;-) T
But what about the functions that exist in them?
Like I said, cookie molds. You use the mold to press cookies, but only the cookies are served, not the mold. The mold may be very complicated, containing subcookies attached to bigger cookies, but whatever is pressed (i.e., instantiated) is what's served on the dinner plate. The mold always remains in the kitchen.
 template A()
 {
     void A()
     {
         B();
     }
     void B() { }
 }
 
 is everything in the template removed 100% or is there junk that the
 compiler doesn't remove?
There is nothing to remove. If you instantiated the template, then you get a copy of everything in it. The number of copies equals the number of distinct instantiations. The template itself is just a mold, an abstract entity that doesn't exist in binary form. What it does is to serve as a mold (hence, "template") to make code. So if you use to make 10 copies of the code, that's what you'll get in your executable. If your template contains 5 functions, then each instantiation produces 5 copies of those functions. Simple as that. Of course, not all code produces binary data -- enum and alias definitions don't produce any binary code, for example -- they're just logical entities that only exist at compile time. So if you have an enum inside a template, it will get copied however many times you instantiate the template, but none of those copies end up in the executable because they're just declarations, not actual code or data.
 Oh... and I'm not talking about theoretical... I'm talking about
 what dmd actually does.
See for yourself: run a disassembler on the result (preferably filtered through ddemangle so the symbols are actually readable) and see what's included and what's not. On Posix, you can use objdump or nm (if you use nm, though, keep in mind that not all symbols correspond with actual code/data, but they do take up a little space in the executable -- at least enough to store their names). T -- MACINTOSH: Most Applications Crash, If Not, The Operating System Hangs
Aug 01 2013
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 1 August 2013 at 18:09:54 UTC, H. S. Teoh wrote:
 See for yourself: run a disassembler on the result (preferably 
 filtered
 through ddemangle so the symbols are actually readable) and see 
 what's
 included and what's not. On Posix, you can use objdump or nm 
 (if you use
 nm, though, keep in mind that not all symbols correspond with 
 actual
 code/data, but they do take up a little space in the executable 
 -- at
 least enough to store their names).
Done it, see my previous comment here. I do have an idea how to fix it, but wasn't able to get though dmd code so far :(
Aug 01 2013
prev sibling parent reply "JS" <js.mdnq gmail.com> writes:
On Thursday, 1 August 2013 at 18:09:54 UTC, H. S. Teoh wrote:
 On Thu, Aug 01, 2013 at 07:52:28PM +0200, JS wrote:
 On Thursday, 1 August 2013 at 17:47:00 UTC, H. S. Teoh wrote:
On Thu, Aug 01, 2013 at 07:12:51PM +0200, John Colvin wrote:
On Thursday, 1 August 2013 at 17:09:07 UTC, JS wrote:
If I have a bunch of templates that are only used for code
generation, are they removed in the binary(since they are 
not
needed)?
templates don't exist in binaries.
Templates are like cookie molds, you use them to generate lots of (almost) identical cookies, but you never serve the mold to the customer. ;-) T
But what about the functions that exist in them?
Like I said, cookie molds. You use the mold to press cookies, but only the cookies are served, not the mold. The mold may be very complicated, containing subcookies attached to bigger cookies, but whatever is pressed (i.e., instantiated) is what's served on the dinner plate. The mold always remains in the kitchen.
 template A()
 {
     void A()
     {
         B();
     }
     void B() { }
 }
 
 is everything in the template removed 100% or is there junk 
 that the
 compiler doesn't remove?
There is nothing to remove. If you instantiated the template, then you get a copy of everything in it. The number of copies equals the number of distinct instantiations. The template itself is just a mold, an abstract entity that doesn't exist in binary form. What it does is to serve as a mold (hence, "template") to make code. So if you use to make 10 copies of the code, that's what you'll get in your executable. If your template contains 5 functions, then each instantiation produces 5 copies of those functions. Simple as that. Of course, not all code produces binary data -- enum and alias definitions don't produce any binary code, for example -- they're just logical entities that only exist at compile time. So if you have an enum inside a template, it will get copied however many times you instantiate the template, but none of those copies end up in the executable because they're just declarations, not actual code or data.
Ok, I'm not talking about the template itself but what is contained in the template. It is obvious that templates can "insert" stuff into the binary. a mixin template can easily do that. Now are you telling me that template A() { void foo() { writeln("asdf"); } } void main() { A!().foo(); } does not create a function foo in the binary? That it is equivalent to just calling writeln("asdf"); directly? (exact same code) e.g., it is the same as void main() { A!().foo(); } cause when I actually debug I see a function call to foo. So, saying that templates are like cookie cutter doesn't prove anything. If I have void main() { } the binary file is is 150kB. If I add import std.stdio; It jumps to 300kB. So the linker and compiler are not removing all untouched code.
Aug 01 2013
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Aug 01, 2013 at 10:06:54PM +0200, JS wrote:
[...]
 Now are you telling me that
 
 template A()
 {
     void foo() { writeln("asdf"); }
 }
 void main()
 {
 	A!().foo();
 }
 
 does not create a function foo in the binary? That it is equivalent
 to just calling writeln("asdf"); directly? (exact same code)
[...] I said that every instantiation of a template creates a copy of everything inside. Therefore, A!().foo() will create a copy of A.foo() in the binary. The template itself has no binary representation, in the sense that if you write: template A(int x) { void foo() { writeln(x); } } there is nothing in the binary corresponding with the template A, or the uninstantiated function foo. But if you instantiate A with some value of x, then you will get a copy of A.foo for every value of x that you instantiate the template with. So if you write: A!1.foo(); A!2.foo(); A!3.foo(); Then you will get 3 copies of foo() in your executable, one for each value of x. T -- What do you get if you drop a piano down a mineshaft? A flat minor.
Aug 01 2013
parent reply "JS" <js.mdnq gmail.com> writes:
On Thursday, 1 August 2013 at 21:17:34 UTC, H. S. Teoh wrote:
 On Thu, Aug 01, 2013 at 10:06:54PM +0200, JS wrote:
 [...]
 Now are you telling me that
 
 template A()
 {
     void foo() { writeln("asdf"); }
 }
 void main()
 {
 	A!().foo();
 }
 
 does not create a function foo in the binary? That it is 
 equivalent
 to just calling writeln("asdf"); directly? (exact same code)
[...] I said that every instantiation of a template creates a copy of everything inside. Therefore, A!().foo() will create a copy of A.foo() in the binary. The template itself has no binary representation, in the sense that if you write: template A(int x) { void foo() { writeln(x); } } there is nothing in the binary corresponding with the template A, or the uninstantiated function foo. But if you instantiate A with some value of x, then you will get a copy of A.foo for every value of x that you instantiate the template with. So if you write: A!1.foo(); A!2.foo(); A!3.foo(); Then you will get 3 copies of foo() in your executable, one for each value of x.
yes, I understand that... now use a template for a string mixin!!!!! template A() { string A() { ... } } ... mixin(A()); IS A GOING TO BE IN THE BINARY?!?!?! Yes, I'm yelling... just to get the point across about the question I'm trying to get answered. the function A is never used at runtime SO it should technically not be in the binary UNLESS dmd treats it as a normal template function then it will(but shouldn't)! e.g., if the compiler smart enough to realize that A(); is different from mixin(A()); (one being compile time and the other not)
Aug 01 2013
next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Thursday, 1 August 2013 at 22:04:57 UTC, JS wrote:
 IS A GOING TO BE IN THE BINARY?!?!?!
Why don't you try it and see? Disassemble the object file or get your linker of choice to print out some info.
Aug 01 2013
parent "JS" <js.mdnq gmail.com> writes:
On Thursday, 1 August 2013 at 22:13:25 UTC, John Colvin wrote:
 On Thursday, 1 August 2013 at 22:04:57 UTC, JS wrote:
 IS A GOING TO BE IN THE BINARY?!?!?!
Why don't you try it and see? Disassemble the object file or get your linker of choice to print out some info.
I don't have any of the tools to do so at the moment. The point was to see if anyone actually knew this specifically or was just guessing on what should be... I guess I have my answer(e.g., find out yourself = I don't know).
Aug 01 2013
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 1 August 2013 at 22:04:57 UTC, JS wrote:
 IS A GOING TO BE IN THE BINARY?!?!?! Yes, I'm yelling... just 
 to get the point across about the question I'm trying to get 
 answered.
HOW MANY TIMES I NEED TO ANSWER YOUR QUESTION FOR YOU TO NOTICE? Yes.
Aug 01 2013
parent reply "JS" <js.mdnq gmail.com> writes:
On Thursday, 1 August 2013 at 22:26:06 UTC, Dicebot wrote:
 On Thursday, 1 August 2013 at 22:04:57 UTC, JS wrote:
 IS A GOING TO BE IN THE BINARY?!?!?! Yes, I'm yelling... just 
 to get the point across about the question I'm trying to get 
 answered.
HOW MANY TIMES I NEED TO ANSWER YOUR QUESTION FOR YOU TO NOTICE? Yes.
UM, NO, BECAUSE YOU ARE NOT ANSWERING MY QUESTION! I don't know why it is so damn confusing... I didn't say templates, I SAID string mixins of templates. I did a test http://dpaste.dzfl.pl/d8e6ca2a http://dpaste.dzfl.pl/a43b923c What is weird, on my comp, the file sizes are exactly the same, searching for the string only shows up in the one with the template call. So It seems the dmd does not add code for string mixins(even though the function is called, it is smart enough not to add it to the binary)... unless it is somehow encrypting the template(again, I would expect the file size to be smaller... unless it does some type of alignment/padding).
Aug 01 2013
parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 1 August 2013 at 22:42:11 UTC, JS wrote:
 I don't know why it is so damn confusing...

 I didn't say templates, I SAID string mixins of templates.
Symbol emitting does not go that way. Object files are old school entities from C times. They have function symbol and type symbols. Templates, mixins - does not matter. If function is used - it goes to object file. If struct definition gets generated - same story. In the world of linker there are no things such as templates or anything like that.
Aug 01 2013
parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 1 August 2013 at 22:59:14 UTC, Dicebot wrote:
 ...
To further clarify this, I have used your simplified example: ------------- // test.d string generator(int x) { if (x > 0) { asm { mov EAX, 0xDEAD; } } return "return 42;"; } int main() { mixin(generator(0)); } ------------- $ dmd -release -inline -O test.d ------------- $ objdump -d test | grep dead 417705: b8 ad de 00 00 mov $0xdead,%eax
Aug 01 2013
parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 1 August 2013 at 23:15:47 UTC, Dicebot wrote:
 On Thursday, 1 August 2013 at 22:59:14 UTC, Dicebot wrote:
 ...
To further clarify this, I have used your simplified example:
And in case someone thinks template functions are treated any differently, "string generator(T)(T a)" acts exactly the same.
Aug 01 2013
prev sibling parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Thursday, 1 August 2013 at 20:06:56 UTC, JS wrote:

 the binary file is is 150kB. If I add import std.stdio; It 
 jumps to 300kB.
This scares me, it seems to me that the unused import should not be compiled in. And I frequently hear people talk about the linker removing unused code, but this proves that it doesn't or at least not all. I wonder if this is a matter of something that is possible but just not implemented properly(or at all, i don't know) in the linker, or if their is some reason that makes this not possible.
Aug 01 2013
parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 1 August 2013 at 21:38:15 UTC, Tofu Ninja wrote:
 This scares me, it seems to me that the unused import should 
 not be compiled in. And I frequently hear people talk about the 
 linker removing unused code, but this proves that it doesn't or 
 at least not all.
It does not. People talk about this because it seems reasonable. But reality differs a lot. No need to ask - just go an check.
Aug 01 2013
prev sibling parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 08/01/2013 10:45 AM, H. S. Teoh wrote:
 On Thu, Aug 01, 2013 at 07:12:51PM +0200, John Colvin wrote:
 On Thursday, 1 August 2013 at 17:09:07 UTC, JS wrote:
 If I have a bunch of templates that are only used for code
 generation, are they removed in the binary(since they are not
 needed)?
templates don't exist in binaries.
Templates are like cookie molds, you use them to generate lots of (almost) identical cookies, but you never serve the mold to the customer. ;-) T
I think the question is whether the instantiations of such templates are removed. If the instance is used only to initialize an enum, the function shouldn't stay in the binary. Even if the function remains, I think the linker takes care of removing unused functions. Ali
Aug 01 2013
next sibling parent reply "JS" <js.mdnq gmail.com> writes:
On Thursday, 1 August 2013 at 17:53:38 UTC, Ali Çehreli wrote:
 On 08/01/2013 10:45 AM, H. S. Teoh wrote:
 On Thu, Aug 01, 2013 at 07:12:51PM +0200, John Colvin wrote:
 On Thursday, 1 August 2013 at 17:09:07 UTC, JS wrote:
 If I have a bunch of templates that are only used for code
 generation, are they removed in the binary(since they are not
 needed)?
templates don't exist in binaries.
Templates are like cookie molds, you use them to generate lots of (almost) identical cookies, but you never serve the mold to the customer. ;-) T
I think the question is whether the instantiations of such templates are removed. If the instance is used only to initialize an enum, the function shouldn't stay in the binary. Even if the function remains, I think the linker takes care of removing unused functions. Ali
I have a lot of code generation templates(only for code). I want to make sure they don't hang around in the binary(even strings they use) as there is no point and it is potentially dangerous(security issues).
Aug 01 2013
parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Thursday, 1 August 2013 at 17:57:28 UTC, JS wrote:
 On Thursday, 1 August 2013 at 17:53:38 UTC, Ali Çehreli wrote:
 On 08/01/2013 10:45 AM, H. S. Teoh wrote:
 On Thu, Aug 01, 2013 at 07:12:51PM +0200, John Colvin wrote:
 On Thursday, 1 August 2013 at 17:09:07 UTC, JS wrote:
 If I have a bunch of templates that are only used for code
 generation, are they removed in the binary(since they are 
 not
 needed)?
templates don't exist in binaries.
Templates are like cookie molds, you use them to generate lots of (almost) identical cookies, but you never serve the mold to the customer. ;-) T
I think the question is whether the instantiations of such templates are removed. If the instance is used only to initialize an enum, the function shouldn't stay in the binary. Even if the function remains, I think the linker takes care of removing unused functions. Ali
I have a lot of code generation templates(only for code). I want to make sure they don't hang around in the binary(even strings they use) as there is no point and it is potentially dangerous(security issues).
The linker will get rid of anything not needed. Marking things as package (or more restrictive) will make sure they don't stay in librarys when not used. Ultimately, if you're worried about it, get the linker to print a list of all functions, or check the assembly.
Aug 01 2013
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 1 August 2013 at 17:53:38 UTC, Ali Çehreli wrote:
 I think the question is whether the instantiations of such 
 templates are removed. If the instance is used only to 
 initialize an enum, the function shouldn't stay in the binary.

 Even if the function remains, I think the linker takes care of 
 removing unused functions.

 Ali
Unfortunately, you are wrong. No symbol optimization is done by DMD is linker can't figure this out on its own, it requires specific object file layout to do reference counting for symbols. It is really easy to check - make template function with some anchor value inside (OXDEADBEEF) and compile it with this template used exclusively in constraints or CTFE. Then strip and objdump - you'll find anchors in resulting binary for every instantiation.
Aug 01 2013