www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Idea for a new tool

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
https://issues.dlang.org/show_bug.cgi?id=5051 -- Andrei
Oct 14 2016
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 15/10/2016 2:13 AM, Andrei Alexandrescu wrote:

 https://issues.dlang.org/show_bug.cgi?id=5051 -- Andrei


As asked for time and time again, this shouldn't be an external tool 
handling it but dmd-fe should.
Sadly this will require some significant work.
Oct 14 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/14/2016 09:15 AM, rikki cattermole wrote:

 On 15/10/2016 2:13 AM, Andrei Alexandrescu wrote:

 https://issues.dlang.org/show_bug.cgi?id=5051 -- Andrei


 As asked for time and time again, this shouldn't be an external tool
 handling it but dmd-fe should.


What is dmd-fe?


 Sadly this will require some significant work.


s/Sadly/Fortunately because now we have smart graduate students looking 
for cool projects/


Andrei
Oct 14 2016
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 15/10/2016 2:17 AM, Andrei Alexandrescu wrote:

 On 10/14/2016 09:15 AM, rikki cattermole wrote:

 On 15/10/2016 2:13 AM, Andrei Alexandrescu wrote:

 https://issues.dlang.org/show_bug.cgi?id=5051 -- Andrei


 As asked for time and time again, this shouldn't be an external tool
 handling it but dmd-fe should.


 What is dmd-fe?


dmd front end
Oct 14 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/14/2016 09:20 AM, rikki cattermole wrote:

 On 15/10/2016 2:17 AM, Andrei Alexandrescu wrote:

 On 10/14/2016 09:15 AM, rikki cattermole wrote:

 On 15/10/2016 2:13 AM, Andrei Alexandrescu wrote:

 https://issues.dlang.org/show_bug.cgi?id=5051 -- Andrei


 As asked for time and time again, this shouldn't be an external tool
 handling it but dmd-fe should.


 What is dmd-fe?


 dmd front end


Yah, making the front end a library would probably be a good way to 
create such a tool. -- Andrei
Oct 14 2016
next sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
On 15/10/2016 2:21 AM, Andrei Alexandrescu wrote:

 On 10/14/2016 09:20 AM, rikki cattermole wrote:

 On 15/10/2016 2:17 AM, Andrei Alexandrescu wrote:

 On 10/14/2016 09:15 AM, rikki cattermole wrote:

 On 15/10/2016 2:13 AM, Andrei Alexandrescu wrote:

 https://issues.dlang.org/show_bug.cgi?id=5051 -- Andrei


 As asked for time and time again, this shouldn't be an external tool
 handling it but dmd-fe should.


 What is dmd-fe?


 dmd front end


 Yah, making the front end a library would probably be a good way to
 create such a tool. -- Andrei


It is to my understanding that to implement such a thing would require a 
significant refactor + additions. There is a reason why it hasn't been 
hacked in yet even with such high demand.

But I'll leave it to somebody more knowledgeable about it to actually 
say what has to be done.
Oct 14 2016
prev sibling parent reply Mathias Lang <mathias.lang sociomantic.com> writes:
On Friday, 14 October 2016 at 13:21:18 UTC, Andrei Alexandrescu 
wrote:

 On 10/14/2016 09:20 AM, rikki cattermole wrote:

 On 15/10/2016 2:17 AM, Andrei Alexandrescu wrote:

 On 10/14/2016 09:15 AM, rikki cattermole wrote:

 On 15/10/2016 2:13 AM, Andrei Alexandrescu wrote:

 https://issues.dlang.org/show_bug.cgi?id=5051 -- Andrei


 As asked for time and time again, this shouldn't be an 
 external tool
 handling it but dmd-fe should.


 What is dmd-fe?


 dmd front end


 Yah, making the front end a library would probably be a good 
 way to create such a tool. -- Andrei


And to enable the creation of many others.

I've looked into pluging DMD-FE inside a Vibe.d server a while 
ago, in order to be able to link usages to declarations (and the 
other way around) using a Github API.

There are a couple of things that prevented any usage of the FE 
as a library:
- The frontend uses loads of globals and static - not only does 
it makes for an horrible library interface, but it makes it so 
that you cannot just discard the result of a semantic analysis 
and start over.

- DMD currently disables the GC. Enabling it causes random errors 
during semantic.

- All files in DMD should have to be moved to match their package 
definition ( explanation in 
https://issues.dlang.org/show_bug.cgi?id=16071 , there was also a 
thread on this topic a couple of weeks ago).

- The frontend depends on the backend when inline ASM is involved 
(and if you want to parse druntime's modules...). You can make 
this a soft dependency in some cases, but then the frontend just 
won't see your `ret` for example, so your flow control should not 
depend on your inline ASM.

- There were also random errors, most likely due to memory 
corruption, caused when compiling the frontend in (without any 
actual usage). For example, a global AA in Vibe.d got it's length 
reset.


While those issues are fixable with some effort, there doesn't 
seem to be any particular interest upstream towards making this 
work. If it is indeed a desirable direction, then maybe we could 
head to dmd-internals about what needs to be done, and how.
Oct 14 2016
parent rikki cattermole <rikki cattermole.co.nz> writes:
On 15/10/2016 4:47 AM, Mathias Lang wrote:

 On Friday, 14 October 2016 at 13:21:18 UTC, Andrei Alexandrescu wrote:

 On 10/14/2016 09:20 AM, rikki cattermole wrote:

 On 15/10/2016 2:17 AM, Andrei Alexandrescu wrote:

 On 10/14/2016 09:15 AM, rikki cattermole wrote:

 On 15/10/2016 2:13 AM, Andrei Alexandrescu wrote:

 https://issues.dlang.org/show_bug.cgi?id=5051 -- Andrei


 As asked for time and time again, this shouldn't be an external tool
 handling it but dmd-fe should.


 What is dmd-fe?


 dmd front end


 Yah, making the front end a library would probably be a good way to
 create such a tool. -- Andrei


 And to enable the creation of many others.

 I've looked into pluging DMD-FE inside a Vibe.d server a while ago, in
 order to be able to link usages to declarations (and the other way
 around) using a Github API.

 There are a couple of things that prevented any usage of the FE as a
 library:
 - The frontend uses loads of globals and static - not only does it makes
 for an horrible library interface, but it makes it so that you cannot
 just discard the result of a semantic analysis and start over.

 - DMD currently disables the GC. Enabling it causes random errors during
 semantic.

 - All files in DMD should have to be moved to match their package
 definition ( explanation in
 https://issues.dlang.org/show_bug.cgi?id=16071 , there was also a thread
 on this topic a couple of weeks ago).

 - The frontend depends on the backend when inline ASM is involved (and
 if you want to parse druntime's modules...). You can make this a soft
 dependency in some cases, but then the frontend just won't see your
 `ret` for example, so your flow control should not depend on your inline
 ASM.


We really should move the asm block into druntime.
A single pragma could replace most of the special behavior considering 
we have CTFE.
Of course to do this properly we need to be able pass "code" as string 
to be replaced and then mixed in.

struct Asm_x86 {
     string __astParse(string text) {}
}

Asm_x86 asm;

void func() {
	asm {
		naked;
		ret;
	}
}

void func() {
	pragma(__ast_naked);
	pragma(__ast_raw, [0xC3]);
}

But I did mention this previously in a reply to another thread, there is 
still holes and needs somebody else to really go through it.
Oct 14 2016
prev sibling parent ketmar <ketmar ketmar.no-ip.org> writes:
On Friday, 14 October 2016 at 13:17:39 UTC, Andrei Alexandrescu 
wrote:

 s/Sadly/Fortunately because now we have smart graduate students 
 looking for cool projects/


...like reimplementing the whole DMDFE for nothing. this tool 
would be useless without full-featured template and CTFE support 
(as most mixins comes from there), so say "hello" to template 
engine, semantic analyser, and CTFE engine (wow!). and then 
you'll probably won't get valid D code again, 'cause for CTFE it 
is easier to lower it to "impossible" AST (like current DMDFE 
does).

btw, it's not hard to print resulting AST in dmd, but at this 
stage it is not valid D code -- and for a reason.
Oct 15 2016
prev sibling next sibling parent reply qznc <qznc web.de> writes:
On Friday, 14 October 2016 at 13:13:16 UTC, Andrei Alexandrescu 
wrote:

 https://issues.dlang.org/show_bug.cgi?id=5051 -- Andrei


Reminds me of Lisp, where they also have code generation wrapped 
in code generation wrapped in ... here is an example: 
http://stackoverflow.com/q/16579844/2361979

Lispers have a REPL and can expand interactively and 
specifically, though.

I guess the most awesome D use case would be inside an IDE. Click 
on a template/mixin call to expand exactly that one. An IDE would 
have access to all build and link parameters, which naturally 
influence this.

For example, click on "foo.find(bar)" and it opens a new 
buffer/tab/window, where the correct variant of find is 
instantiated according to foo and bar.

Click on "mixin Foo;" and it inlines the instantiated code.
Oct 14 2016
parent =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Friday, 14 October 2016 at 13:53:15 UTC, qznc wrote:

 I guess the most awesome D use case would be inside an IDE. 
 Click on a template/mixin call to expand exactly that one. An 
 IDE would have access to all build and link parameters, which 
 naturally influence this.

 For example, click on "foo.find(bar)" and it opens a new 
 buffer/tab/window, where the correct variant of find is 
 instantiated according to foo and bar.

 Click on "mixin Foo;" and it inlines the instantiated code.


I agree, I've been dreaming about this for a long time aswell.
Oct 14 2016
prev sibling next sibling parent reply Chris Wright <dhasenan gmail.com> writes:
On Fri, 14 Oct 2016 09:13:16 -0400, Andrei Alexandrescu wrote:


 https://issues.dlang.org/show_bug.cgi?id=5051 -- Andrei


For the love of potatoes, please take a few seconds to tell us the gist 
of it here.

The requested tool is a preprocessor, of sorts: evaluate version 
statements and mixins and expand templates. That lets you see what's 
going on in your code a bit better in the case of extensive 
metaprogramming.
Oct 14 2016
parent reply Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D
Subject: Re: Idea for a new tool
References: <ntqlld$2c4p$1 digitalmars.com> <ntqrvv$2lan$1 digitalmars.com>
In-Reply-To: <ntqrvv$2lan$1 digitalmars.com>

--a8GHEEOC4pSTvlLnLkaHUTR4d2spE937W
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 10/14/2016 06:01 PM, Chris Wright wrote:

 On Fri, 14 Oct 2016 09:13:16 -0400, Andrei Alexandrescu wrote:
=20

 https://issues.dlang.org/show_bug.cgi?id=3D5051 -- Andrei


=20
 For the love of potatoes, please take a few seconds to tell us the gist=


=20

 of it here.
=20
 The requested tool is a preprocessor, of sorts: evaluate version=20
 statements and mixins and expand templates. That lets you see what's=20
 going on in your code a bit better in the case of extensive=20
 metaprogramming.


This wouldn't be a pre-processor in C/C++ sense because it requires full
semantic analysis to work. More like a special mode of compilation.


--a8GHEEOC4pSTvlLnLkaHUTR4d2spE937W--
Oct 14 2016
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/14/2016 12:26 PM, Dicebot wrote:

 On 10/14/2016 06:01 PM, Chris Wright wrote:

 On Fri, 14 Oct 2016 09:13:16 -0400, Andrei Alexandrescu wrote:


 https://issues.dlang.org/show_bug.cgi?id=5051 -- Andrei


 For the love of potatoes, please take a few seconds to tell us the gist
 of it here.

 The requested tool is a preprocessor, of sorts: evaluate version
 statements and mixins and expand templates. That lets you see what's
 going on in your code a bit better in the case of extensive
 metaprogramming.


 This wouldn't be a pre-processor in C/C++ sense because it requires full
 semantic analysis to work. More like a special mode of compilation.


Yep - a "lowerer" :o). -- Andrei
Oct 14 2016
prev sibling next sibling parent =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Friday, 14 October 2016 at 13:13:16 UTC, Andrei Alexandrescu 
wrote:

 https://issues.dlang.org/show_bug.cgi?id=5051 -- Andrei


I have tons of stuff I would like to help out fixing in DMD, 
including this.

IMO, the large obstacle against making these happen isn't the 
technical challenge but lack of documentation and forum support 
when working on these issues. If each kernel developer set aside 
just 10 minutes each day in the forums giving hints on where to 
start digging in DMD to help out implementing all the things 
people want in D, we would attract more developers who can help 
out improved diagnostics, fixing bugs, IDE support like this. 
Soon those time investments will feedback to all D developers in 
improved productivity and stability. By some called "the law of 
accelerating returns".

I have a personal wishlist for improvements in the D compiler and 
some ideas on how to implement them. Is there a public common 
place where I can append these? Then we could link DMD 
documentation from the issues of that list.

In other words:

Ask not what D can do you, but instead
Ask what you can do for D!

Can we please make this happen?
Oct 14 2016
prev sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Friday, 14 October 2016 at 13:13:16 UTC, Andrei Alexandrescu 
wrote:

 https://issues.dlang.org/show_bug.cgi?id=5051 -- Andrei


I have been working on this.
What this amounts to is an AST-writeout using the HdrGen.
This will affect the code in so far as it will be lowerd my the 
dmd frontend.

I have hit problems with massive template recursion.
I am working on ways to fix it.
(with a __symbol pseudo type which allows to turn recursion into 
iteration, and removes the significant template-instanciation 
overhead)


 Nordlöw
Looking at this PR is very good place to start.
https://github.com/dlang/dmd/pull/426
It should be rather easily portable to ddmd.
Oct 14 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/14/16 12:38 PM, Stefan Koch wrote:

 On Friday, 14 October 2016 at 13:13:16 UTC, Andrei Alexandrescu wrote:

 https://issues.dlang.org/show_bug.cgi?id=5051 -- Andrei


 I have been working on this.
 What this amounts to is an AST-writeout using the HdrGen.
 This will affect the code in so far as it will be lowerd my the dmd
 frontend.

 I have hit problems with massive template recursion.
 I am working on ways to fix it.
 (with a __symbol pseudo type which allows to turn recursion into
 iteration, and removes the significant template-instanciation overhead)


This is awesome! Do you think you could coordinate with Martin and one 
of the new students to divide effort on that? Would be great if you 
stayed focused on CTFE jitting. Thanks! -- Andrei
Oct 14 2016
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Friday, 14 October 2016 at 18:02:24 UTC, Andrei Alexandrescu 
wrote:

 On 10/14/16 12:38 PM, Stefan Koch wrote:

 On Friday, 14 October 2016 at 13:13:16 UTC, Andrei 
 Alexandrescu wrote:

 https://issues.dlang.org/show_bug.cgi?id=5051 -- Andrei


 I have been working on this.
 What this amounts to is an AST-writeout using the HdrGen.
 This will affect the code in so far as it will be lowerd my 
 the dmd
 frontend.

 I have hit problems with massive template recursion.
 I am working on ways to fix it.
 (with a __symbol pseudo type which allows to turn recursion 
 into
 iteration, and removes the significant template-instanciation 
 overhead)


 This is awesome! Do you think you could coordinate with Martin 
 and one of the new students to divide effort on that? Would be 
 great if you stayed focused on CTFE jitting. Thanks! -- Andrei


I can certainly give pointers to anyone who wishes to take this 
issue.
Oct 14 2016