www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Request for Review: DI Generation Improvements

reply "Adam Wilson" <flyboynw gmail.com> writes:
I am formally requesting review of the two pull requests that make up my
DI Generation Improvements patch.

The first pull is against the DRuntime and modifies the makefiles such
some files in the DRT are not generated as DI files but copied from the D
source files directly into the DI file. This solves a problem where
certain portions of Phobos the CTFE into the DRT cause a build failure
when using the new DI generation. The files that are copied instead of
generated are all of core and core.stdc, EXCEPT core.atomic, core.cpuid,
core.demangle, and core.thread.

Pull DRuntime/210:
https://github.com/D-Programming-Language/druntime/pull/210

The second pull is against DMD itself and consists of the actual changes
to the current DI generation. The improvements consist of removing all
function implementations, EXCEPT for those of template and auto-return
functions, and improved indenting. These changes represent the closest I
can come to agreement from the community as to the proper contents of a DI
file. This patch fixes the biggest problems of DI generation and provides
a baseline for future improvements to DI generation. Currently being
discussed are modifications to include a limited form of semantic analysis
of the source code during DI generation to improve the usefulness of the
output as well as the addition of an attribute to direct DI generation to
include a function implementation for CTFE or inlining purposes.

Pull DMD/945: https://github.com/D-Programming-Language/dmd/pull/945

I welcome all comments on these pulls and it is my goal to see them merged
into DMD 2.060.

-- 
Adam Wilson
IRC: LightBender
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
May 14 2012
next sibling parent reply =?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <xtzgzorex gmail.com> writes:
On 15-05-2012 01:50, Adam Wilson wrote:
 I am formally requesting review of the two pull requests that make up my
 DI Generation Improvements patch.

 The first pull is against the DRuntime and modifies the makefiles such
 some files in the DRT are not generated as DI files but copied from the D
 source files directly into the DI file. This solves a problem where
 certain portions of Phobos the CTFE into the DRT cause a build failure
 when using the new DI generation. The files that are copied instead of
 generated are all of core and core.stdc, EXCEPT core.atomic, core.cpuid,
 core.demangle, and core.thread.

 Pull DRuntime/210:
 https://github.com/D-Programming-Language/druntime/pull/210

 The second pull is against DMD itself and consists of the actual changes
 to the current DI generation. The improvements consist of removing all
 function implementations, EXCEPT for those of template and auto-return
 functions, and improved indenting. These changes represent the closest I
 can come to agreement from the community as to the proper contents of a DI
 file. This patch fixes the biggest problems of DI generation and provides
 a baseline for future improvements to DI generation. Currently being
 discussed are modifications to include a limited form of semantic analysis
 of the source code during DI generation to improve the usefulness of the
 output as well as the addition of an attribute to direct DI generation to
 include a function implementation for CTFE or inlining purposes.

 Pull DMD/945: https://github.com/D-Programming-Language/dmd/pull/945

 I welcome all comments on these pulls and it is my goal to see them merged
 into DMD 2.060.

I would suggest excluding core.[atomic,cpuid,demangle] from DI generation, too, and just copy them. I don't see any reason to do DI generation for them. -- - Alex
May 14 2012
next sibling parent =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <xtzgzorex gmail.com> writes:
On 15-05-2012 02:20, Tyro[17] wrote:
 On Tuesday, 15 May 2012 at 00:02:23 UTC, Alex Rønne Petersen wrote:
 On 15-05-2012 01:50, Adam Wilson wrote:
 I am formally requesting review of the two pull requests that make up my
 DI Generation Improvements patch.

 The first pull is against the DRuntime and modifies the makefiles such
 some files in the DRT are not generated as DI files but copied from
 the D
 source files directly into the DI file. This solves a problem where
 certain portions of Phobos the CTFE into the DRT cause a build failure
 when using the new DI generation. The files that are copied instead of
 generated are all of core and core.stdc, EXCEPT core.atomic, core.cpuid,
 core.demangle, and core.thread.


 I would suggest excluding core.[atomic,cpuid,demangle] from DI
 generation, too, and just copy them. I don't see any reason to do DI
 generation for them.

Reread the last line of the paragraph starting with "The first pull". Your concern is already addressed.

No, you got it backwards. Before my comment, his pull request still caused DI generation to happen for the modules I mentioned, and I suggested simply copying them like the rest. -- - Alex
May 14 2012
prev sibling parent =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <xtzgzorex gmail.com> writes:
On 15-05-2012 02:59, Jonathan M Davis wrote:
 On Monday, May 14, 2012 17:36:49 Adam Wilson wrote:
 This can be easily changed. Does anyone see any potential problems with
 making this change?

The only modules in druntime that I am aware of where there would be actual problems if their .d files were used instead of .di files would be those which already have hand-maintained .di files (core.thread and object are the only two that I'm aware of, though there might be others). Beyond that, core.rt definitely should be using .di files, but AFAIK, it wouldn't actually hurt anything beyond compilation times for them to be .d files, and AFAIK, all of the rest of druntime can be .d files without causing any problems. Most of them just hold C declarations anyway. - Jonathan M Davis

core.rt? Do you mean rt.*? Those are completely and entirely private anyway. -- - Alex
May 14 2012
prev sibling next sibling parent "Tyro[17]" <nospam home.com> writes:
On Tuesday, 15 May 2012 at 00:02:23 UTC, Alex Rønne Petersen 
wrote:
 On 15-05-2012 01:50, Adam Wilson wrote:
 I am formally requesting review of the two pull requests that 
 make up my
 DI Generation Improvements patch.

 The first pull is against the DRuntime and modifies the 
 makefiles such
 some files in the DRT are not generated as DI files but copied 
 from the D
 source files directly into the DI file. This solves a problem 
 where
 certain portions of Phobos the CTFE into the DRT cause a build 
 failure
 when using the new DI generation. The files that are copied 
 instead of
 generated are all of core and core.stdc, EXCEPT core.atomic, 
 core.cpuid,
 core.demangle, and core.thread.


 I would suggest excluding core.[atomic,cpuid,demangle] from DI 
 generation, too, and just copy them. I don't see any reason to 
 do DI generation for them.

Reread the last line of the paragraph starting with "The first pull". Your concern is already addressed.
May 14 2012
prev sibling next sibling parent "Adam Wilson" <flyboynw gmail.com> writes:
On Mon, 14 May 2012 17:02:23 -0700, Alex R=F8nne Petersen
<xtzgzorex gmail.com> wrote:

 On 15-05-2012 01:50, Adam Wilson wrote:
 I am formally requesting review of the two pull requests that make up=


 DI Generation Improvements patch.

 The first pull is against the DRuntime and modifies the makefiles suc=


 some files in the DRT are not generated as DI files but copied from t=


 D
 source files directly into the DI file. This solves a problem where
 certain portions of Phobos the CTFE into the DRT cause a build failur=


 when using the new DI generation. The files that are copied instead o=


 generated are all of core and core.stdc, EXCEPT core.atomic, core.cpu=


 core.demangle, and core.thread.

 Pull DRuntime/210:
 https://github.com/D-Programming-Language/druntime/pull/210

 The second pull is against DMD itself and consists of the actual chan=


 to the current DI generation. The improvements consist of removing al=


 function implementations, EXCEPT for those of template and auto-retur=


 functions, and improved indenting. These changes represent the closes=


 can come to agreement from the community as to the proper contents of=


 DI
 file. This patch fixes the biggest problems of DI generation and  =


 provides
 a baseline for future improvements to DI generation. Currently being
 discussed are modifications to include a limited form of semantic  =


 analysis
 of the source code during DI generation to improve the usefulness of =


 output as well as the addition of an attribute to direct DI generatio=


 to
 include a function implementation for CTFE or inlining purposes.

 Pull DMD/945: https://github.com/D-Programming-Language/dmd/pull/945

 I welcome all comments on these pulls and it is my goal to see them  =


 merged
 into DMD 2.060.

I would suggest excluding core.[atomic,cpuid,demangle] from DI =

 generation, too, and just copy them. I don't see any reason to do DI  =

 generation for them.

This can be easily changed. Does anyone see any potential problems with making this change? -- = Adam Wilson IRC: LightBender Project Coordinator The Horizon Project http://www.thehorizonproject.org/
May 14 2012
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Monday, May 14, 2012 17:36:49 Adam Wilson wrote:
 This can be easily changed. Does anyone see any potential problems with
 making this change?

The only modules in druntime that I am aware of where there would be actual problems if their .d files were used instead of .di files would be those which already have hand-maintained .di files (core.thread and object are the only two that I'm aware of, though there might be others). Beyond that, core.rt definitely should be using .di files, but AFAIK, it wouldn't actually hurt anything beyond compilation times for them to be .d files, and AFAIK, all of the rest of druntime can be .d files without causing any problems. Most of them just hold C declarations anyway. - Jonathan M Davis
May 14 2012
prev sibling next sibling parent "Tyro[17]" <nospam home.com> writes:
On Tuesday, 15 May 2012 at 00:25:15 UTC, Alex Rønne Petersen
wrote:
 On 15-05-2012 02:20, Tyro[17] wrote:
 On Tuesday, 15 May 2012 at 00:02:23 UTC, Alex Rønne Petersen


 I would suggest excluding core.[atomic,cpuid,demangle] from DI
 generation, too, and just copy them. I don't see any reason 
 to do DI
 generation for them.

Reread the last line of the paragraph starting with "The first pull". Your concern is already addressed.

No, you got it backwards. Before my comment, his pull request still caused DI generation to happen for the modules I mentioned, and I suggested simply copying them like the rest.

I see. Pardon the noise.
May 14 2012
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Tuesday, May 15, 2012 03:20:24 Alex R=C3=B8nne Petersen wrote:
 On 15-05-2012 02:59, Jonathan M Davis wrote:
 On Monday, May 14, 2012 17:36:49 Adam Wilson wrote:
 This can be easily changed. Does anyone see any potential problems=



 making this change?

The only modules in druntime that I am aware of where there would b=


 actual
 problems if their .d files were used instead of .di files would be =


 which already have hand-maintained .di files (core.thread and objec=


 the only two that I'm aware of, though there might be others). Beyo=


 that, core.rt definitely should be using .di files, but AFAIK, it
 wouldn't actually hurt anything beyond compilation times for them t=


 .d files, and AFAIK, all of the rest of druntime can be .d files wi=


 causing any problems. Most of them just hold C declarations anyway.=


=20
 - Jonathan M Davis

core.rt? Do you mean rt.*? Those are completely and entirely private =

Probably. I didn't look at the source tree before posting. My point was= that=20 the rt stuff shouldn't be provided as .d files (however they may be org= anized),=20 whereas pretty much everything other than core.thread and object should= be fine=20 as .d files AFAIK. - Jonathan M Davis
May 14 2012
prev sibling next sibling parent reply "Adam Wilson" <flyboynw gmail.com> writes:
On Mon, 14 May 2012 16:50:24 -0700, Adam Wilson <flyboynw gmail.com> wrote:

 I am formally requesting review of the two pull requests that make up my
 DI Generation Improvements patch.

 The first pull is against the DRuntime and modifies the makefiles such
 some files in the DRT are not generated as DI files but copied from the D
 source files directly into the DI file. This solves a problem where
 certain portions of Phobos the CTFE into the DRT cause a build failure
 when using the new DI generation. The files that are copied instead of
 generated are all of core and core.stdc, EXCEPT core.atomic, core.cpuid,
 core.demangle, and core.thread.

 Pull DRuntime/210:
 https://github.com/D-Programming-Language/druntime/pull/210

 The second pull is against DMD itself and consists of the actual changes
 to the current DI generation. The improvements consist of removing all
 function implementations, EXCEPT for those of template and auto-return
 functions, and improved indenting. These changes represent the closest I
 can come to agreement from the community as to the proper contents of a  
 DI
 file. This patch fixes the biggest problems of DI generation and provides
 a baseline for future improvements to DI generation. Currently being
 discussed are modifications to include a limited form of semantic  
 analysis
 of the source code during DI generation to improve the usefulness of the
 output as well as the addition of an attribute to direct DI generation to
 include a function implementation for CTFE or inlining purposes.

 Pull DMD/945: https://github.com/D-Programming-Language/dmd/pull/945

 I welcome all comments on these pulls and it is my goal to see them  
 merged
 into DMD 2.060.

I have updated the make files so that only core.thread and core.sync.* are run through DI generation. ALL other core.* modules are copied into the import directory now. Git wrecked my branch on a rebase so I had to recreate the pull. Pull DRuntime/218: https://github.com/D-Programming-Language/druntime/pull/218 -- Adam Wilson IRC: LightBender Project Coordinator The Horizon Project http://www.thehorizonproject.org/
May 14 2012
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 06/11/2012 09:37 AM, timotheecour wrote:
 questions:

 A) as I understand it, the new di generation will systematically strip out the
implementation of
 non-auto-return, non-templated functions, is that correct?

This is my understanding as well.
B) since there are some important differences with the current di files (in
terms of inlining
 optimization, etc), will there be a dmd command-line flag to output those
stripped down di files
 (eg: -stripdi), so user still has choice of which to output ?

You could use cp instead of dmd -H.
 C) why can't auto-return functions be semantically analyzed and resolved?
 (eg:auto fun(int x){return x;} ) => generated di should be: int fun(int
 x); )

Conditional compilation. version(A) int x; else version(B) double x; else static assert(0); auto foo(){return x;} would need to be stripped to version(A){ int x; int foo(); }else version(B){ double x; double foo(); }else static assert(0); which is a nontrivial transformation. This is just a simple example. Resolving the return type conditionally in the general case is undecidable, therefore, making it work satisfactorily involves a potentially unbounded amount of work.
 D) can we have an option to strip out template functions as well? This
 could be more or less customizable, eg with a file that contains a list
 of template functions to strip, or simply strip all templates). The
 library writer would instantiate those templates to certain predefined
 values. Eg:

 module fun;
 T fun1(T)(T x){
      return 2*x;
 }
 void dummy_instantiate(){
 //instantiate to double, and repeat for all desired types, eg with a mixin.
      alias double T;
      fun1!(T)(T.init);
 }
 Then the library writer generates a library (static or shared) and the
 end user uses the templates with one of the allowed types (otherwise
 link error happens). In many cases (eg matrix/numerical libraries), all
 that's needed is a finite set of predefined types (eg int,float etc).
 Bonus points would be if the generated di file automatically generates
 template constraints to reflect the allowed types, to have compile time
 errors instead of link time ones.
 Having ability to strip templates greatly simplifies distribution of
 code, as it doesn't have to carry all dependencies recursively if all
 one wants is a few predefined types.

You could use overloads instead and use templates for implementing them. Templates are generally only exposed in a well-designed library interface if they work with an unbounded number of types.
 D) btw, is there a way to force the instantiations more elegantly rather
 than using dummy_instantiate? in C++ we can just write something like:
 template double fun<double>(double);, but the same doesnt in D.

For example: T foo(T)(T arg){return arg; pragma(msg, T);} mixin template Instantiate(alias t,T...){ static assert({ void _(){mixin t!T;} return true; }()); } mixin Instantiate!(foo,int); mixin Instantiate!(foo,double); The nested function named '_' is unnecessary. It works around a DMD bug, mixin t!T is 'not yet implemented in CTFE'.
Jun 11 2012
prev sibling next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Monday, May 14, 2012 23:03:20 Adam Wilson wrote:
 I have updated the make files so that only core.thread and core.sync.* are
 run through DI generation. ALL other core.* modules are copied into the
 import directory now.

I assume that object.di and core/thread.di are being used rather than their corresponding .d files being run through the .di generation? They already have hand-crafted .di files. - Jonathan M Davis
May 14 2012
next sibling parent =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <alex lycus.org> writes:
On 16-05-2012 14:46, Steven Schveighoffer wrote:
 On Tue, 15 May 2012 05:51:44 -0400, kenji hara <k.hara.pg gmail.com> wrote:

 Old days import/core/thread.di was generated from src/core/thread.d .
 Current import/core/thread.di is generated from src/core/thread.*di* .

Huh? Why the copy? Just move src/core/thread.di to import/core/thread.di object.di lives in import/core, I think it should be the same for all the hand-maintained .di files.

Good point. I wonder why we didn't do that initially.
 FWIW, I thought thread.di was being generated because of this.

 Also, I agree that thread and object are the only modules that need to
 be .di files. Everything else is already opaque for the most part, and
 the pieces that aren't are just supporting code that can be visible.

 What we need to protect is the runtime implementation, so projects
 cannot depend on private APIs that may change.

 -Steve

-- Alex Rønne Petersen alex lycus.org http://lycus.org
May 16 2012
prev sibling parent =?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:
On 16-05-2012 19:00, Adam Wilson wrote:
 On Wed, 16 May 2012 05:46:44 -0700, Steven Schveighoffer
 <schveiguy yahoo.com> wrote:

 On Tue, 15 May 2012 05:51:44 -0400, kenji hara <k.hara.pg gmail.com>
 wrote:

 Old days import/core/thread.di was generated from src/core/thread.d .
 Current import/core/thread.di is generated from src/core/thread.*di* .

Huh? Why the copy? Just move src/core/thread.di to import/core/thread.di object.di lives in import/core, I think it should be the same for all the hand-maintained .di files.

Actually, object.di lives /import. Moving thread.di to /import/core would require a significant change to the clean targets of both makefiles since they just rm the whole /import/core directory at the moment. It's not impossible and if you guys agree that the change should be made I'd be happy to make it happen.

I think it's a good idea.
 FWIW, I thought thread.di was being generated because of this.

 Also, I agree that thread and object are the only modules that need to
 be .di files. Everything else is already opaque for the most part, and
 the pieces that aren't are just supporting code that can be visible.

 What we need to protect is the runtime implementation, so projects
 cannot depend on private APIs that may change.

 -Steve

The biggest problem right now is that, while we all agree that these changes need to happen, getting them merged appears to be nigh impossible. There appears to be a bottleneck in the process caused by the lack of capable persons to verify and execute the merges, particularly for druntime. There are 11 passing merges for the druntime at last count, out of 28 total. Alexrp and dawgfoto particularly seem to have a lot of pulls that are just languishing.

-- Alex Rnne Petersen alex lycus.org http://lycus.org
May 16 2012
prev sibling next sibling parent "Adam Wilson" <flyboynw gmail.com> writes:
On Mon, 14 May 2012 23:11:50 -0700, Jonathan M Davis <jmdavisProg gmx.com>  
wrote:

 On Monday, May 14, 2012 23:03:20 Adam Wilson wrote:
 I have updated the make files so that only core.thread and core.sync.*  
 are
 run through DI generation. ALL other core.* modules are copied into the
 import directory now.

I assume that object.di and core/thread.di are being used rather than their corresponding .d files being run through the .di generation? They already have hand-crafted .di files. - Jonathan M Davis

The funny thing is that core.thread.di was being run through the DI generator in the old make file. I had left it that way, I have posted a commit that moves it to the copy section. -- Adam Wilson IRC: LightBender Project Coordinator The Horizon Project http://www.thehorizonproject.org/
May 14 2012
prev sibling next sibling parent kenji hara <k.hara.pg gmail.com> writes:
Old days import/core/thread.di was generated from src/core/thread.d .
Current import/core/thread.di is generated from src/core/thread.*di* .

Kenji Hara

2012/5/15 Jonathan M Davis <jmdavisProg gmx.com>:
 On Monday, May 14, 2012 23:46:08 Adam Wilson wrote:
 On Mon, 14 May 2012 23:11:50 -0700, Jonathan M Davis <jmdavisProg gmx.com>

 wrote:
 On Monday, May 14, 2012 23:03:20 Adam Wilson wrote:
 I have updated the make files so that only core.thread and core.sync.*
 are
 run through DI generation. ALL other core.* modules are copied into the
 import directory now.

I assume that object.di and core/thread.di are being used rather than their corresponding .d files being run through the .di generation? They already have hand-crafted .di files. - Jonathan M Davis

The funny thing is that core.thread.di was being run through the DI generator in the old make file. I had left it that way, I have posted a commit that moves it to the copy section.

Well, if that works... It's weird though, because there's no reason to have a thread.di if it's not used, and as I understand it, some work was being done that required some of Thread's internals _not_ be in the .di file, but I don't remember the details, and I don't know where that stands. - Jonathan M Davis

May 15 2012
prev sibling next sibling parent "Adam Wilson" <flyboynw gmail.com> writes:
On Tue, 15 May 2012 02:51:44 -0700, kenji hara <k.hara.pg gmail.com> wrote:

 Old days import/core/thread.di was generated from src/core/thread.d .
 Current import/core/thread.di is generated from src/core/thread.*di* .

 Kenji Hara

Well, I've updated the make file to just copy thread.di instead of running it through the generator. -- Adam Wilson IRC: LightBender Project Coordinator The Horizon Project http://www.thehorizonproject.org/
May 15 2012
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 15 May 2012 05:51:44 -0400, kenji hara <k.hara.pg gmail.com> wrote:

 Old days import/core/thread.di was generated from src/core/thread.d .
 Current import/core/thread.di is generated from src/core/thread.*di* .

Huh? Why the copy? Just move src/core/thread.di to import/core/thread.di object.di lives in import/core, I think it should be the same for all the hand-maintained .di files. FWIW, I thought thread.di was being generated because of this. Also, I agree that thread and object are the only modules that need to be .di files. Everything else is already opaque for the most part, and the pieces that aren't are just supporting code that can be visible. What we need to protect is the runtime implementation, so projects cannot depend on private APIs that may change. -Steve
May 16 2012
prev sibling next sibling parent "Adam Wilson" <flyboynw gmail.com> writes:
On Wed, 16 May 2012 05:46:44 -0700, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:

 On Tue, 15 May 2012 05:51:44 -0400, kenji hara <k.hara.pg gmail.com>  
 wrote:

 Old days import/core/thread.di was generated from src/core/thread.d .
 Current import/core/thread.di is generated from src/core/thread.*di* .

Huh? Why the copy? Just move src/core/thread.di to import/core/thread.di object.di lives in import/core, I think it should be the same for all the hand-maintained .di files.

Actually, object.di lives /import. Moving thread.di to /import/core would require a significant change to the clean targets of both makefiles since they just rm the whole /import/core directory at the moment. It's not impossible and if you guys agree that the change should be made I'd be happy to make it happen.
 FWIW, I thought thread.di was being generated because of this.

 Also, I agree that thread and object are the only modules that need to  
 be .di files.  Everything else is already opaque for the most part, and  
 the pieces that aren't are just supporting code that can be visible.

 What we need to protect is the runtime implementation, so projects  
 cannot depend on private APIs that may change.

 -Steve

The biggest problem right now is that, while we all agree that these changes need to happen, getting them merged appears to be nigh impossible. There appears to be a bottleneck in the process caused by the lack of capable persons to verify and execute the merges, particularly for druntime. There are 11 passing merges for the druntime at last count, out of 28 total. Alexrp and dawgfoto particularly seem to have a lot of pulls that are just languishing. -- Adam Wilson IRC: LightBender Project Coordinator The Horizon Project http://www.thehorizonproject.org/
May 16 2012
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Wednesday, May 16, 2012 10:00:24 Adam Wilson wrote:
 The biggest problem right now is that, while we all agree that these
 changes need to happen, getting them merged appears to be nigh impossible.
 There appears to be a bottleneck in the process caused by the lack of
 capable persons to verify and execute the merges, particularly for
 druntime. There are 11 passing merges for the druntime at last count, out
 of 28 total. Alexrp and dawgfoto particularly seem to have a lot of pulls
 that are just languishing.

That's just life with druntime, Phobos, and dmd. There are only so many people with commit access, and they're all busy. druntime is a bit worse than Phobos, since it often requires Sean to look at stuff, but they both have the problem. And in this particular case, since it impacts so much, it's one of the requests that Sean probabyl should look at, and I don't know how often he goes through druntime's pull requests. - Jonathan M Davis
May 16 2012
prev sibling next sibling parent "Adam Wilson" <flyboynw gmail.com> writes:
On Wed, 16 May 2012 14:37:46 -0700, Jonathan M Davis <jmdavisProg gmx.com>  
wrote:

 On Wednesday, May 16, 2012 10:00:24 Adam Wilson wrote:
 The biggest problem right now is that, while we all agree that these
 changes need to happen, getting them merged appears to be nigh  
 impossible.
 There appears to be a bottleneck in the process caused by the lack of
 capable persons to verify and execute the merges, particularly for
 druntime. There are 11 passing merges for the druntime at last count,  
 out
 of 28 total. Alexrp and dawgfoto particularly seem to have a lot of  
 pulls
 that are just languishing.

That's just life with druntime, Phobos, and dmd. There are only so many people with commit access, and they're all busy. druntime is a bit worse than Phobos, since it often requires Sean to look at stuff, but they both have the problem. And in this particular case, since it impacts so much, it's one of the requests that Sean probabyl should look at, and I don't know how often he goes through druntime's pull requests. - Jonathan M Davis

I absolutely agree that he should look at it. I was just kvetching about the backlog in general. There are some really good fixes that aren't getting merged. Realistically, I no longer expect for the full DI patch to make it in to 2.060; but at the same time I don't want it to get forgotten, and there are A LOT of bitrotten pulls on GitHub. -- Adam Wilson IRC: LightBender Project Coordinator The Horizon Project http://www.thehorizonproject.org/
May 16 2012
prev sibling next sibling parent "Adam Wilson" <flyboynw gmail.com> writes:
On Wed, 16 May 2012 05:46:44 -0700, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:

 On Tue, 15 May 2012 05:51:44 -0400, kenji hara <k.hara.pg gmail.com>  
 wrote:

 Old days import/core/thread.di was generated from src/core/thread.d .
 Current import/core/thread.di is generated from src/core/thread.*di* .

Huh? Why the copy? Just move src/core/thread.di to import/core/thread.di object.di lives in import/core, I think it should be the same for all the hand-maintained .di files. FWIW, I thought thread.di was being generated because of this. Also, I agree that thread and object are the only modules that need to be .di files. Everything else is already opaque for the most part, and the pieces that aren't are just supporting code that can be visible. What we need to protect is the runtime implementation, so projects cannot depend on private APIs that may change. -Steve

I have a waiting commit that will move thread.di to import/core and updates the makefiles accordingly, however, I need Zor's threading pulls merged before this can happen. https://github.com/D-Programming-Language/druntime/pull/214/ https://github.com/D-Programming-Language/druntime/pull/204/ -- Adam Wilson IRC: LightBender Project Coordinator The Horizon Project http://www.thehorizonproject.org/
May 16 2012
prev sibling next sibling parent "dawg" <dawg dawgfoto.de> writes:
On Tuesday, 15 May 2012 at 06:46:58 UTC, Adam Wilson wrote:
 On Mon, 14 May 2012 23:11:50 -0700, Jonathan M Davis 
 <jmdavisProg gmx.com> wrote:

 On Monday, May 14, 2012 23:03:20 Adam Wilson wrote:
 I have updated the make files so that only core.thread and 
 core.sync.* are
 run through DI generation. ALL other core.* modules are 
 copied into the
 import directory now.

I assume that object.di and core/thread.di are being used rather than their corresponding .d files being run through the .di generation? They already have hand-crafted .di files. - Jonathan M Davis

The funny thing is that core.thread.di was being run through the DI generator in the old make file. I had left it that way, I have posted a commit that moves it to the copy section.

I did that so that the same di generation logic is applied to the handwritten import files. Currently I don't really expect any difference though.
May 21 2012
prev sibling next sibling parent "Adam Wilson" <flyboynw gmail.com> writes:
On Mon, 21 May 2012 05:12:32 -0700, dawg <dawg dawgfoto.de> wrote:

 On Tuesday, 15 May 2012 at 06:46:58 UTC, Adam Wilson wrote:
 On Mon, 14 May 2012 23:11:50 -0700, Jonathan M Davis  
 <jmdavisProg gmx.com> wrote:

 On Monday, May 14, 2012 23:03:20 Adam Wilson wrote:
 I have updated the make files so that only core.thread and  
 core.sync.* are
 run through DI generation. ALL other core.* modules are copied into  
 the
 import directory now.

I assume that object.di and core/thread.di are being used rather than their corresponding .d files being run through the .di generation? They already have hand-crafted .di files. - Jonathan M Davis

The funny thing is that core.thread.di was being run through the DI generator in the old make file. I had left it that way, I have posted a commit that moves it to the copy section.

I did that so that the same di generation logic is applied to the handwritten import files. Currently I don't really expect any difference though.

Currently, there isn't any difference, however, once DI generation is changed to strip out function implementations it could change significantly. There is even talk of building a limited amount of semantic analysis into DI generation. Because of this, relying on the current DI generation in the future would not be a good idea; these changes are in preparation for future DI changes. -- Adam Wilson IRC: LightBender Project Coordinator The Horizon Project http://www.thehorizonproject.org/
May 21 2012
prev sibling next sibling parent "timotheecour" <thelastmammoth gmail.com> writes:
questions:

A) as I understand it, the new di generation will systematically 
strip out the implementation of non-auto-return, non-templated 
functions, is that correct?

B) since there are some important differences with the current di 
files (in terms of inlining optimization, etc), will there be a 
dmd command-line flag to output those stripped down di files (eg: 
-stripdi), so user still has choice of which to output ?

C) why can't auto-return functions be semantically analyzed and 
resolved?
(eg:auto fun(int x){return x;} ) => generated di should be: int 
fun(int x); )

D) can we have an option to strip out template functions as well? 
This could be more or less customizable, eg with a file that 
contains a list of template functions to strip, or simply strip 
all templates). The library writer would instantiate those 
templates to certain predefined values. Eg:

module fun;
T fun1(T)(T x){
	return 2*x;
}
void dummy_instantiate(){
//instantiate to double, and repeat for all desired types, eg 
with a mixin.
	alias double T;
	fun1!(T)(T.init);
}
Then the library writer generates a library (static or shared) 
and the end user uses the templates with one of the allowed types 
(otherwise link error happens). In many cases (eg 
matrix/numerical libraries), all that's needed is a finite set of 
predefined types (eg int,float etc). Bonus points would be if the 
generated di file automatically generates template constraints to 
reflect the allowed types, to have compile time errors instead of 
link time ones.
Having ability to strip templates greatly simplifies distribution 
of code, as it doesn't have to carry all dependencies recursively 
if all one wants is a few predefined types.

D) btw, is there a way to force the instantiations more elegantly 
rather than using dummy_instantiate? in C++ we can just write 
something like: template double fun<double>(double);, but the 
same doesnt in D.
Jun 11 2012
prev sibling next sibling parent "Adam Wilson" <flyboynw gmail.com> writes:
On Mon, 11 Jun 2012 04:55:37 -0700, Timon Gehr <timon.gehr gmx.ch> wrote:

 On 06/11/2012 09:37 AM, timotheecour wrote:
 questions:

 A) as I understand it, the new di generation will systematically strip  
 out the implementation of
 non-auto-return, non-templated functions, is that correct?

This is my understanding as well.

Correct. A lot of community consultation went into the improvements.
 B) since there are some important differences with the current di files  
 (in terms of inlining
 optimization, etc), will there be a dmd command-line flag to output  
 those stripped down di files
 (eg: -stripdi), so user still has choice of which to output ?

You could use cp instead of dmd -H.

In fact I rewrote the DRuntime makefiles to do precisely this with the hand-crafted .DI files. https://github.com/D-Programming-Language/druntime/pull/218
 C) why can't auto-return functions be semantically analyzed and  
 resolved?
 (eg:auto fun(int x){return x;} ) => generated di should be: int fun(int
 x); )

Conditional compilation. version(A) int x; else version(B) double x; else static assert(0); auto foo(){return x;} would need to be stripped to version(A){ int x; int foo(); }else version(B){ double x; double foo(); }else static assert(0); which is a nontrivial transformation. This is just a simple example. Resolving the return type conditionally in the general case is undecidable, therefore, making it work satisfactorily involves a potentially unbounded amount of work.

The general explanation is that any time you rewrite the AST (such as the operation performed above) you have to duplicate that work in DI generation to maintain semantic cohesion (what I put in is what I get out). Another reason is that DI generation is *required* to be run prior to the semantic analysis stage due to fact that your command line could alter the analysis and subsequent AST. In fact this note is one of the *very* few multi-line comments Walter put into the DMD source. In essence DI generation is an AST pretty-printer, and as such is must be run prior to analysis and after parsing. All my patch does is insert checks into the printing process to stop it from printing certain parts of the AST. Theoretically one could rebuild a semantic analyzer that didn't change it's behavior based on the the command-line into the DI generation, but that would literally require rewriting DI generation from the ground up. And you'd still have to verify that the primary semantic analysis didn't change anything from the DI analysis. Then you'd have to write a reconciliation process. Personally, I think that we have better places to focus our efforts. That said, what you really want is the full source embedded into the library (similar, to .NET's CIL). That would get you want you are after. Such a thing could actually be done except for OMF/Optlink. Since OMF doesn't support custom sections there is a no special place to store the code that the compiler could easily access. This would enable the compiler to extract the source during compilation and analyze the both the user source and library source and perform all possible optimizations accordingly. I look at adding COFF to DMD and my brain melted. There are enough #IFDEFs in there to cause permanent insanity... *sigh*
 D) can we have an option to strip out template functions as well? This
 could be more or less customizable, eg with a file that contains a list
 of template functions to strip, or simply strip all templates). The
 library writer would instantiate those templates to certain predefined
 values. Eg:

 module fun;
 T fun1(T)(T x){
      return 2*x;
 }
 void dummy_instantiate(){
 //instantiate to double, and repeat for all desired types, eg with a  
 mixin.
      alias double T;
      fun1!(T)(T.init);
 }
 Then the library writer generates a library (static or shared) and the
 end user uses the templates with one of the allowed types (otherwise
 link error happens). In many cases (eg matrix/numerical libraries), all
 that's needed is a finite set of predefined types (eg int,float etc).
 Bonus points would be if the generated di file automatically generates
 template constraints to reflect the allowed types, to have compile time
 errors instead of link time ones.
 Having ability to strip templates greatly simplifies distribution of
 code, as it doesn't have to carry all dependencies recursively if all
 one wants is a few predefined types.

You could use overloads instead and use templates for implementing them. Templates are generally only exposed in a well-designed library interface if they work with an unbounded number of types.
 D) btw, is there a way to force the instantiations more elegantly rather
 than using dummy_instantiate? in C++ we can just write something like:
 template double fun<double>(double);, but the same doesnt in D.

For example: T foo(T)(T arg){return arg; pragma(msg, T);} mixin template Instantiate(alias t,T...){ static assert({ void _(){mixin t!T;} return true; }()); } mixin Instantiate!(foo,int); mixin Instantiate!(foo,double); The nested function named '_' is unnecessary. It works around a DMD bug, mixin t!T is 'not yet implemented in CTFE'.

-- Adam Wilson IRC: LightBender Project Coordinator The Horizon Project http://www.thehorizonproject.org/
Jun 11 2012
prev sibling next sibling parent "timotheecour" <thelastmammoth gmail.com> writes:
 mixin Instantiate!(foo,int);

 You could use cp instead of dmd -H.

stripped currently), but I guess the current behavior is relatively useless so it's fine.
 want you are after. Such a thing could actually be done except
 for OMF/Optlink. Since OMF doesn't support custom sections
 there is a no special place to store the code that the compiler
 could easily access.

If we want to embed the AST inside an "import" library file (which is optional, could be done with a directory), wouldn't it be possible to store the AST as a global / static variable? When dmd compiles myfun.d, it generates the AST and inserts a global variable (eg void*_ast_myfun=...) in the data segment of the library/object file (that could be done with a mixin and be portable). just started a new thread for related ideas: http://forum.dlang.org/thread/lmepufogzaxlbxwgubvl forum.dlang.org thanks!
Jun 12 2012
prev sibling parent "Adam Wilson" <flyboynw gmail.com> writes:
On Tue, 12 Jun 2012 02:16:19 -0700, timotheecour  
<thelastmammoth gmail.com> wrote:

 mixin Instantiate!(foo,int);

 You could use cp instead of dmd -H.

stripped currently), but I guess the current behavior is relatively useless so it's fine.
 want you are after. Such a thing could actually be done except
 for OMF/Optlink. Since OMF doesn't support custom sections
 there is a no special place to store the code that the compiler
 could easily access.

If we want to embed the AST inside an "import" library file (which is optional, could be done with a directory), wouldn't it be possible to store the AST as a global / static variable? When dmd compiles myfun.d, it generates the AST and inserts a global variable (eg void*_ast_myfun=...) in the data segment of the library/object file (that could be done with a mixin and be portable). just started a new thread for related ideas: http://forum.dlang.org/thread/lmepufogzaxlbxwgubvl forum.dlang.org thanks!

Theoretically yes, but it's extremely poor form for the compiler to be adding variables to programmer created structures especially when said variables are only ever going to be used by the compiler. -- Adam Wilson IRC: LightBender Project Coordinator The Horizon Project http://www.thehorizonproject.org/
Jun 12 2012