www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Need clarification on dmd symbol generation

reply "Dicebot" <m.strashun gmail.com> writes:
I am currently investigating 
http://d.puremagic.com/issues/show_bug.cgi?id=9571 and after 
brief exploration of symbols emitted to object file as well as 
code that emits them I can't rid of an idea that I am missing 
something about module system.

Why does dmd emits horde of symbols from imported modules into 
object files, most of whose are not actually used in this one? No 
"-inline" used and I kind of expected to see there only directly 
called function and symbols from their signatures.

What does actually happen here?
Apr 04 2013
next sibling parent "Dicebot" <m.strashun gmail.com> writes:
On Thursday, 4 April 2013 at 19:34:06 UTC, Dicebot wrote:
 I am currently investigating 
 http://d.puremagic.com/issues/show_bug.cgi?id=9571 and after 
 brief exploration of symbols emitted to object file as well as 
 code that emits them I can't rid of an idea that I am missing 
 something about module system.

 Why does dmd emits horde of symbols from imported modules into 
 object files, most of whose are not actually used in this one? 
 No "-inline" used and I kind of expected to see there only 
 directly called function and symbols from their signatures.

 What does actually happen here?

Okay, after some digging on the topic I am quite sure that DMD does add instantiated template symbols from imported modules to root module symbol list (template.c:5023). It does not care if root module was actually one to start template instantiation chain (and thus uses this symbol) and just bloats the symbol table. Repeating initial question - was there rationale for this other than being reluctant to store additional context information about initial template chain instantiation scope? I'd like to make a pull request for this and make one step closer to making separate compilation reality but don't want to waste time on dmd source code (argh) without real need, thus preliminary question.
Apr 05 2013
prev sibling next sibling parent "Dicebot" <m.strashun gmail.com> writes:
I am really gonna rise that up until I get some answer.
"GTFO" is acceptable answer and still better than no answer.
Apr 08 2013
prev sibling next sibling parent "Dicebot" <m.strashun gmail.com> writes:
.
Apr 10 2013
prev sibling next sibling parent kenji hara <k.hara.pg gmail.com> writes:
--047d7b62292665132504d9fda20c
Content-Type: text/plain; charset=UTF-8

Sorry I have no answer for the issue. That is still unknown area to me...

Kenji Hara


2013/4/5 Dicebot <m.strashun gmail.com>

 I am currently investigating http://d.puremagic.com/issues/**
 show_bug.cgi?id=9571 <http://d.puremagic.com/issues/show_bug.cgi?id=9571>and
after brief exploration of symbols emitted to object file as well as
 code that emits them I can't rid of an idea that I am missing something
 about module system.

 Why does dmd emits horde of symbols from imported modules into object
 files, most of whose are not actually used in this one? No "-inline" used
 and I kind of expected to see there only directly called function and
 symbols from their signatures.

 What does actually happen here?

--047d7b62292665132504d9fda20c Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Sorry I have no answer for the issue. That is still unknow= n area to me...<br><div><br></div><div>Kenji Hara</div></div><div class=3D"= gmail_extra"><br><br><div class=3D"gmail_quote">2013/4/5 Dicebot <span dir= =3D"ltr">&lt;<a href=3D"mailto:m.strashun gmail.com" target=3D"_blank">m.st= rashun gmail.com</a>&gt;</span><br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex">I am currently investigating <a href=3D"http= ://d.puremagic.com/issues/show_bug.cgi?id=3D9571" target=3D"_blank">http://= d.puremagic.com/issues/<u></u>show_bug.cgi?id=3D9571</a> and after brief ex= ploration of symbols emitted to object file as well as code that emits them= I can&#39;t rid of an idea that I am missing something about module system= .<br> <br> Why does dmd emits horde of symbols from imported modules into object files= , most of whose are not actually used in this one? No &quot;-inline&quot; u= sed and I kind of expected to see there only directly called function and s= ymbols from their signatures.<br> <br> What does actually happen here?<br> </blockquote></div><br></div> --047d7b62292665132504d9fda20c--
Apr 10 2013
prev sibling next sibling parent "Dicebot" <m.strashun gmail.com> writes:
On Wednesday, 10 April 2013 at 08:41:35 UTC, kenji hara wrote:
 Sorry I have no answer for the issue. That is still unknown 
 area to me...

 Kenji Hara

Thanks for replying! Really appreciated. May be you can give me a small insight on dmd internal data representation then? I think I have got the issue and know what needs to be done, but can't find reasonably short solution within existing code structure.
Apr 10 2013
prev sibling next sibling parent kenji hara <k.hara.pg gmail.com> writes:
--f46d0444813110ef4b04d9fe6bda
Content-Type: text/plain; charset=UTF-8

2013/4/10 Dicebot <m.strashun gmail.com>

 On Wednesday, 10 April 2013 at 08:41:35 UTC, kenji hara wrote:

 Sorry I have no answer for the issue. That is still unknown area to me...

 Kenji Hara

Thanks for replying! Really appreciated. May be you can give me a small insight on dmd internal data representation then? I think I have got the issue and know what needs to be done, but can't find reasonably short solution within existing code structure.

OK. I don't understand the issue enough, but I'd like to help you. - TemplateInstance::tinst may represent the enclosing template instance that 'this' instance is instantiated. It is set in TemplateInstance::semantic(|2|3) for its member's semantic process. - TemplateInstance::enclosing may represent the parent of nested template instance. If a template is instantiated in function scope with implicit context, it points the function. It is set in TemplateInstance::hasNestedArgs. - Module::importedFrom may represents the "root module" of import chain. it points one of the module that listed in command line. Good luck. Kenji Hara --f46d0444813110ef4b04d9fe6bda Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">2013/4/10 Dicebot <span dir=3D"ltr">&lt;<a href=3D"mailto:= m.strashun gmail.com" target=3D"_blank">m.strashun gmail.com</a>&gt;</span>= <br><div class=3D"gmail_extra"><div class=3D"gmail_quote"><blockquote class= =3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;bo= rder-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> <div class=3D"im">On Wednesday, 10 April 2013 at 08:41:35 UTC, kenji hara w= rote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"> Sorry I have no answer for the issue. That is still unknown area to me...<b= r> <br> Kenji Hara<br> </blockquote> <br></div> Thanks for replying! Really appreciated.<br> <br> May be you can give me a small insight on dmd internal data representation = then? I think I have got the issue and know what needs to be done, but can&= #39;t find reasonably short solution within existing code structure.<br> </blockquote></div><br></div><div class=3D"gmail_extra" style><div class=3D= "gmail_extra">OK. I don&#39;t understand the issue enough, but I&#39;d like= to help you.</div><div class=3D"gmail_extra"><br></div><div class=3D"gmail= _extra"> - TemplateInstance::tinst may represent the enclosing template instance tha= t &#39;this&#39; instance is instantiated. It is set in TemplateInstance::s= emantic(|2|3) for its member&#39;s semantic process.</div><div class=3D"gma= il_extra"> <br></div><div class=3D"gmail_extra">- TemplateInstance::enclosing may repr= esent the parent of nested template instance. If a template is instantiated= in function scope with implicit context, it points the function. It is set= in TemplateInstance::hasNestedArgs.</div> <div class=3D"gmail_extra"><br></div><div class=3D"gmail_extra">- Module::i= mportedFrom may represents the &quot;root module&quot; of import chain. it = points one of the module that listed in command line.</div><div class=3D"gm= ail_extra"> <br></div><div class=3D"gmail_extra">Good luck.<br></div><div class=3D"gmai= l_extra"><br></div><div class=3D"gmail_extra">Kenji Hara<br></div></div></d= iv> --f46d0444813110ef4b04d9fe6bda--
Apr 10 2013
prev sibling next sibling parent "Dicebot" <m.strashun gmail.com> writes:
On Wednesday, 10 April 2013 at 09:37:44 UTC, kenji hara wrote:
 OK. I don't understand the issue enough, but I'd like to help 
 you.

 - TemplateInstance::tinst may represent the enclosing template 
 instance
 that 'this' instance is instantiated. It is set in
 TemplateInstance::semantic(|2|3) for its member's semantic 
 process.

 - TemplateInstance::enclosing may represent the parent of 
 nested template
 instance. If a template is instantiated in function scope with 
 implicit
 context, it points the function. It is set in
 TemplateInstance::hasNestedArgs.

 - Module::importedFrom may represents the "root module" of 
 import chain. it
 points one of the module that listed in command line.

 Good luck.

 Kenji Hara

I see no TemplateInstance::enclosing, only Scope::enclosing and TemplateInstance is not Scope descendant. Essentially I need to emit symbols not for "importedFrom" module but for module of top-most tinst in chain. I supposed that iterating tinst can do the trick, but it is not. For example, when map is used, it call templated range functions somewhere down the line and iterating up from those don't get you initial map call scope. (Same issue in printInstantiationTrace as it uses this method). Is this a problem with setting correct tinst initialization in other code, my misunderstanding or, probably, this info may be relied upon only in semantic3, not semantic1?
Apr 10 2013
prev sibling next sibling parent "Robert" <jfanatiker gmx.at> writes:
TemplateInstance::enclosing, also does not get you the 
instantiating context, referring to the comment in the source 
code, I think for my tests it was null.

importedFrom was no solution either. The only thing that worked 
for me was walking up with tinst and then use loc. This way you 
get the instantiating file. (Got that from 
printInstantiationTrace) I think it should be possible to use 
this found file name to find the module by iterating 
Module::modules, but I have not even bothered trying, because I 
felt there has to be some better way, unfortunately I also got no 
replies.

At least it helps that I am not alone with this problem :-)

Maybe it is not entirely unlikely that there is in fact no better 
way, as apart from "where do I put the instantiated template" and 
error messages there is really no need to know where the template 
got instantiated.


Best regards,
Robert
Apr 10 2013
prev sibling next sibling parent "Dicebot" <m.strashun gmail.com> writes:
On Wednesday, 10 April 2013 at 11:53:51 UTC, Robert wrote:
 TemplateInstance::enclosing, also does not get you the 
 instantiating context, referring to the comment in the source 
 code, I think for my tests it was null.

 importedFrom was no solution either. The only thing that worked 
 for me was walking up with tinst and then use loc. This way you 
 get the instantiating file. (Got that from 
 printInstantiationTrace) I think it should be possible to use 
 this found file name to find the module by iterating 
 Module::modules, but I have not even bothered trying, because I 
 felt there has to be some better way, unfortunately I also got 
 no replies.

 At least it helps that I am not alone with this problem :-)

 Maybe it is not entirely unlikely that there is in fact no 
 better way, as apart from "where do I put the instantiated 
 template" and error messages there is really no need to know 
 where the template got instantiated.

Excited to see I am not the only one caring about this! :) Module* can be found via tinst->scope->module, so it is pretty straightforward. I have tried to use this and it _almost_ worked but some symbols were still missing (I have mentioned example with map and range before, can reduce test case if you are interested). importFrom is obvious hack instead of precisely tracking symbol source - "lets just emit weak symbols for every compiled file from all his imports and let linker clean up the mess". Considering Kenji's explanation this seems to work as I have initially understood. Then I need to reduce that case and see why those std.range template symbols are not propagated to the top of the chain. This is a major blocker for getting reliable and efficient separate compilation and bugzilla issue references in topic is just an observable side-effect.
Apr 10 2013
prev sibling next sibling parent kenji hara <k.hara.pg gmail.com> writes:
--089e013c6c0e984d2804da00ea15
Content-Type: text/plain; charset=UTF-8

2013/4/10 Dicebot <m.strashun gmail.com>

 On Wednesday, 10 April 2013 at 09:37:44 UTC, kenji hara wrote:

 OK. I don't understand the issue enough, but I'd like to help you.

 - TemplateInstance::tinst may represent the enclosing template instance
 that 'this' instance is instantiated. It is set in
 TemplateInstance::semantic(|2|**3) for its member's semantic process.

 - TemplateInstance::enclosing may represent the parent of nested template
 instance. If a template is instantiated in function scope with implicit
 context, it points the function. It is set in
 TemplateInstance::**hasNestedArgs.

 - Module::importedFrom may represents the "root module" of import chain.
 it
 points one of the module that listed in command line.

 Good luck.

 Kenji Hara

I see no TemplateInstance::enclosing, only Scope::enclosing and TemplateInstance is not Scope descendant.

Very recently TemplateInstance::isnested is renamed to TemplateInstance::enclosing.
 Essentially I need to emit symbols not for "importedFrom" module but for
 module of top-most tinst in chain. I supposed that iterating tinst can do
 the trick, but it is not. For example, when map is used, it call templated
 range functions somewhere down the line and iterating up from those don't
 get you initial map call scope. (Same issue in printInstantiationTrace as
 it uses this method).

It sounds that using TemplateInstance::enclosing is correct. Kenji Hara --089e013c6c0e984d2804da00ea15 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">2013/4/10 Dicebot <span dir=3D"ltr">&lt;<a href=3D"mailto:= m.strashun gmail.com" target=3D"_blank">m.strashun gmail.com</a>&gt;</span>= <br><div class=3D"gmail_extra"><div class=3D"gmail_quote"><blockquote class= =3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;bo= rder-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> <div class=3D"im">On Wednesday, 10 April 2013 at 09:37:44 UTC, kenji hara w= rote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"> OK. I don&#39;t understand the issue enough, but I&#39;d like to help you.<= br> <br> - TemplateInstance::tinst may represent the enclosing template instance<br> that &#39;this&#39; instance is instantiated. It is set in<br> TemplateInstance::semantic(|2|<u></u>3) for its member&#39;s semantic proce= ss.<br> <br> - TemplateInstance::enclosing may represent the parent of nested template<b= r> instance. If a template is instantiated in function scope with implicit<br> context, it points the function. It is set in<br> TemplateInstance::<u></u>hasNestedArgs.<br> <br> - Module::importedFrom may represents the &quot;root module&quot; of import= chain. it<br> points one of the module that listed in command line.<br> <br> Good luck.<br> <br> Kenji Hara<br> </blockquote> <br></div> I see no TemplateInstance::enclosing, only Scope::enclosing and TemplateIns= tance is not Scope descendant.</blockquote><div><br></div><div style>Very r= ecently TemplateInstance::isnested is renamed to TemplateInstance::enclosin= g.</div> <div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px = 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-l= eft-style:solid;padding-left:1ex">Essentially I need to emit symbols not fo= r &quot;importedFrom&quot; module but for module of top-most tinst in chain= . I supposed that iterating tinst can do the trick, but it is not. For exam= ple, when map is used, it call templated range functions somewhere down the= line and iterating up from those don&#39;t get you initial map call scope.= (Same issue in printInstantiationTrace as it uses this method).<br> </blockquote><div><br></div><div style>It sounds that using TemplateInstanc= e::enclosing is correct.</div><div style><br></div><div style>Kenji Hara</d= iv></div></div></div> --089e013c6c0e984d2804da00ea15--
Apr 10 2013
prev sibling next sibling parent "Dicebot" <m.strashun gmail.com> writes:
On Wednesday, 10 April 2013 at 12:36:35 UTC, kenji hara wrote:
 ...

Can't really express how big my gratitude is! :) Already trying "enclosing".
Apr 10 2013
prev sibling next sibling parent "Dicebot" <m.strashun gmail.com> writes:
You now what? It worked.

Before:
$ nm a.o | wc -l
358

After:
$ nm a.o | wc -l
68

I'll run a full test suite and do the pull request.

Kenji you are my hero ;)
Apr 10 2013
prev sibling next sibling parent Robert <jfanatiker gmx.at> writes:
 Excited to see I am not the only one caring about this! :) 
 Module* can be found via tinst->scope->module, so it is pretty 
 straightforward. I have tried to use this and it _almost_ worked 
 but some symbols were still missing (I have mentioned example 
 with map and range before, can reduce test case if you are 
 interested).

For me tinst->scope->module resolved to the module where the template is declared not the instantiating one, while tinst->loc.filename is the instantiating file. I am interested!
 
 Considering Kenji's explanation this seems to work as I have 
 initially understood. Then I need to reduce that case and see why 
 those std.range template symbols are not propagated to the top of 
 the chain.
 
 This is a major blocker for getting reliable and efficient 
 separate compilation and bugzilla issue references in topic is 
 just an observable side-effect.

Getting reliable building is also quite important for dub, I don't want to work around dmd bugs (sometimes not even feasible), I would rather fix them :-)
Apr 10 2013
prev sibling next sibling parent "Dicebot" <m.strashun gmail.com> writes:
On Wednesday, 10 April 2013 at 14:15:11 UTC, Robert wrote:
 ...

Yes, have found some issues with scope->module and finally resorted to getModule(). Please follow pull request : https://github.com/D-Programming-Language/dmd/pull/1882
Apr 10 2013
prev sibling next sibling parent Robert <jfanatiker gmx.at> writes:
On Wed, 2013-04-10 at 15:50 +0200, Dicebot wrote:
 You now what? It worked.
 
 Before:
 $ nm a.o | wc -l
 358
 
 After:
 $ nm a.o | wc -l
 68
 
 I'll run a full test suite and do the pull request.
 
 Kenji you are my hero ;)

:-( Still wrong output for my code. I commented on your pull request with more details.
Apr 10 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 10 April 2013 at 15:16:31 UTC, Robert wrote:
 On Wed, 2013-04-10 at 15:50 +0200, Dicebot wrote:
 You now what? It worked.
 
 Before:
 $ nm a.o | wc -l
 358
 
 After:
 $ nm a.o | wc -l
 68
 
 I'll run a full test suite and do the pull request.
 
 Kenji you are my hero ;)

:-( Still wrong output for my code. I commented on your pull request with more details.

Please add sample code to http://d.puremagic.com/issues/show_bug.cgi?id=9571
Apr 10 2013
prev sibling parent "Dicebot" <m.strashun gmail.com> writes:
On Wednesday, 10 April 2013 at 16:01:08 UTC, deadalnix wrote:
 Please add sample code to 
 http://d.puremagic.com/issues/show_bug.cgi?id=9571

I know what Robert is speaking about, it is different issue though also related to template symbols. It does not belong to 9571 but I try to address it too.
Apr 10 2013