www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - D source code formatting utility (dformat)

reply Walter Bright <newshound2 digitalmars.com> writes:
Continuing in the vein of the Exceptional coding style thread, D needs (and has 
needed for a long time) a utility that will reformat source code.

One use of it would be to run as a filter on all of the Druntime and Phobos 
source code before checkin.

This could make great use of the current projects to implement a D lexer and 
parser in D.

It'd make for a nice boon to the D community.

Any champions for this?
Jan 15 2013
next sibling parent "Brian Schott" <briancschott gmail.com> writes:
On Tuesday, 15 January 2013 at 23:50:12 UTC, Walter Bright wrote:

 Continuing in the vein of the Exceptional coding style thread, 
 D needs (and has needed for a long time) a utility that will 
 reformat source code.

 One use of it would be to run as a filter on all of the 
 Druntime and Phobos source code before checkin.

 This could make great use of the current projects to implement 
 a D lexer and parser in D.

 It'd make for a nice boon to the D community.

 Any champions for this?



Uncrustify[1] had a release on the first of this month which I'm 
sure includes the fix I made to the parser to recognize the => 
lamba syntax. It may be easier to just use that instead of making 
our own. (I haven't tested to see what it does with user-defined 
attributes yet)

On the other hand, if we need a utility written in D, I'd like to 
finish work on my lexer so that it can be phobos-quality first.

[1] http://sourceforge.net/projects/uncrustify/
Jan 15 2013
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Tuesday, January 15, 2013 15:50:10 Walter Bright wrote:

 Continuing in the vein of the Exceptional coding style thread, D needs (and
 has needed for a long time) a utility that will reformat source code.
 
 One use of it would be to run as a filter on all of the Druntime and Phobos
 source code before checkin.



Honestly, I would hate to have anything like this run on druntime or Phobos. 
We've specifically avoided enforcing exact formatting rules in the style guide 
and instead have focused on the API. In addition, code formatters always end 
up mangling code. Sure, some of the code looks fine, but other parts of it 
inevitably look hideous, because the formatter can never deal with special 
cases well enough. You just can't make the formatting rules flexible enough, 
and sometimes, you just plain have to break formatting guidelines in order to 
make code legible. I have no problem with anyone writing a code formatter, but 
I sure don't want to deal with a code base that uses one.


 This could make great use of the current projects to implement a D lexer and
 parser in D.



I really need to get back to this, but life is so hectic...

- Jonathan M Davis
Jan 15 2013
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/15/13 6:50 PM, Walter Bright wrote:

 Continuing in the vein of the Exceptional coding style thread, D needs
 (and has needed for a long time) a utility that will reformat source code.

 One use of it would be to run as a filter on all of the Druntime and
 Phobos source code before checkin.

 This could make great use of the current projects to implement a D lexer
 and parser in D.

 It'd make for a nice boon to the D community.

 Any champions for this?



What about just using the json output?

Andrei
Jan 15 2013
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/15/2013 4:45 PM, Andrei Alexandrescu wrote:

 What about just using the json output?



It'd lose all the comments.
Jan 15 2013
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/15/13 7:56 PM, Andrej Mitrovic wrote:

 On 1/16/13, Walter Bright<newshound2 digitalmars.com>  wrote:

 On 1/15/2013 4:45 PM, Andrei Alexandrescu wrote:

 What about just using the json output?



 It'd lose all the comments.



 Json could output comments if we wanted to.

 This isn't such a bad idea considering the amount of work necessary to
 actually parse D code in the first place.



Agreed. Generally we should standardize tooling on json, regardless of 
the notion that indeed a parser is good in the library. Walter?

Andrei
Jan 15 2013
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/15/2013 4:56 PM, Andrej Mitrovic wrote:

 This isn't such a bad idea considering the amount of work necessary to
 actually parse D code in the first place.




It isn't just the comments. Suppose you have the value in source code:

    0x100

and then in the json code, it's

    256u

This kind of thing is way beyond what json is intended for. json is there to 
extract semantic content, not syntactic.
Jan 15 2013
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/15/2013 5:24 PM, Andrej Mitrovic wrote:

 I'm pretty sure Walter specifically wants this written in D. C/C++
 solutions already exist, but a D implementation could be a good way to
 show how much work is required to write a code formatter in D and
 could e.g. be presented in DConf. String processing should be D's
 powerful side, after all.



It's also an excellent way to prove the value of the nascent lexer & parser 
being written in D.
Jan 15 2013
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 1/16/13, Walter Bright <newshound2 digitalmars.com> wrote:

 On 1/15/2013 4:45 PM, Andrei Alexandrescu wrote:

 What about just using the json output?



 It'd lose all the comments.



Json could output comments if we wanted to.

This isn't such a bad idea considering the amount of work necessary to
actually parse D code in the first place.
Jan 15 2013
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 16 January 2013 at 00:49:21 UTC, Walter Bright 
wrote:

 On 1/15/2013 4:45 PM, Andrei Alexandrescu wrote:

 What about just using the json output?



 It'd lose all the comments.



And the code itself, unless I'm missing some new enhancement to 
the json... the best you can make from that is prototypes (and 
even that is incomplete right now!)

It seems to me that the easiest way to do this might be to do 
something similar to .di generation, inside the compiler. All the 
code is there, but we'd have to be sure to do do it before any of 
the lowerings done in semantic; if the user writes "foreach", we 
don't want the code formatter spitting out "for".
Jan 15 2013
prev sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 1/16/13, Adam D. Ruppe <destructionator gmail.com> wrote:

 And the code itself, unless I'm missing some new enhancement to
 the json...



Good point.

On 1/16/13, Adam D. Ruppe <destructionator gmail.com> wrote:

 It seems to me that the easiest way to do this might be to do
 something similar to .di generation, inside the compiler.



Note the title!

I'm pretty sure Walter specifically wants this written in D. C/C++
solutions already exist, but a D implementation could be a good way to
show how much work is required to write a code formatter in D and
could e.g. be presented in DConf. String processing should be D's
powerful side, after all.
Jan 15 2013
prev sibling next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 01/16/2013 12:50 AM, Walter Bright wrote:

 Continuing in the vein of the Exceptional coding style thread, D needs
 (and has needed for a long time) a utility that will reformat source code.

 One use of it would be to run as a filter on all of the Druntime and
 Phobos source code before checkin.

 This could make great use of the current projects to implement a D lexer
 and parser in D.

 It'd make for a nice boon to the D community.

 Any champions for this?



It is on my TODO list.
Jan 15 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-01-16 00:50, Walter Bright wrote:

 Continuing in the vein of the Exceptional coding style thread, D needs
 (and has needed for a long time) a utility that will reformat source code.

 One use of it would be to run as a filter on all of the Druntime and
 Phobos source code before checkin.

 This could make great use of the current projects to implement a D lexer
 and parser in D.

 It'd make for a nice boon to the D community.

 Any champions for this?



Now where's that lexer/parser ...

-- 
/Jacob Carlborg
Jan 15 2013
next sibling parent reply "qznc" <qznc go.to> writes:
On Wednesday, 16 January 2013 at 07:40:24 UTC, Jacob Carlborg 
wrote:

 On 2013-01-16 00:50, Walter Bright wrote:

 Continuing in the vein of the Exceptional coding style thread, 
 D needs
 (and has needed for a long time) a utility that will reformat 
 source code.

 One use of it would be to run as a filter on all of the 
 Druntime and
 Phobos source code before checkin.

 This could make great use of the current projects to implement 
 a D lexer
 and parser in D.

 It'd make for a nice boon to the D community.

 Any champions for this?



 Now where's that lexer/parser ...



I think Dil would be a good starting point. It claims 99% of 
lexer/parser done.

https://github.com/azizk/dil
Jan 15 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-01-16 08:54, qznc wrote:


 I think Dil would be a good starting point. It claims 99% of
 lexer/parser done.

 https://github.com/azizk/dil



I always forget about that one. Do we have a list of lexers/parsers 
available?

-- 
/Jacob Carlborg
Jan 16 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-01-16 11:50, Philippe Sigaud wrote:


 I have a small (woefully incomplete) list in my Pegged parser
 generator project README:

 Parser generators:

 * Hisayuki Mima's [CTPG](https://github.com/youkei/ctpg), very
 similar, also done in D. Have a look!

 * Nick Sabalausky's [Goldie](http://www.dsource.org/projects/goldie).

 Parsers:

 * Benjamin Shropshire's
 [dparser](http://dsource.org/projects/scrapple/browser/trunk/dparser).

 * Martin Nowak put these gists on the D newsgroup:
      - https://gist.github.com/1255439 - lexer generator
      - https://gist.github.com/1262321 - complete and fast D lexer


 I should also add the parser from Dil and the one from SDC (At least,
 I suppose there is one in SDC)



That's a good start. We should put these on the wiki.


 Jacob, do you have a link to your parser?



I don't have a parser. That's why I'm complaining all the time that we 
should have a lexer and parser for D written in D :)


 Also, Timon seems to have a parser, and JM Davis was working on a D lexer.




-- 
/Jacob Carlborg
Jan 16 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-01-16 17:41, Philippe Sigaud wrote:


 OK, I'll do that tonight. That will become a reflex ;)



Great :)

-- 
/Jacob Carlborg
Jan 16 2013
prev sibling next sibling parent Philippe Sigaud <philippe.sigaud gmail.com> writes:
 * Martin Nowak put these gists on the D newsgroup:
     - https://gist.github.com/1255439 - lexer generator
     - https://gist.github.com/1262321 - complete and fast D lexer



New link:

https://github.com/dawgfoto/lexer
Jan 16 2013
prev sibling next sibling parent Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Wed, Jan 16, 2013 at 11:33 AM, Jacob Carlborg <doob me.com> wrote:

 On 2013-01-16 08:54, qznc wrote:


 I think Dil would be a good starting point. It claims 99% of
 lexer/parser done.

 https://github.com/azizk/dil




 I always forget about that one. Do we have a list of lexers/parsers
 available?



I have a small (woefully incomplete) list in my Pegged parser
generator project README:

Parser generators:

* Hisayuki Mima's [CTPG](https://github.com/youkei/ctpg), very
similar, also done in D. Have a look!

* Nick Sabalausky's [Goldie](http://www.dsource.org/projects/goldie).

Parsers:

* Benjamin Shropshire's
[dparser](http://dsource.org/projects/scrapple/browser/trunk/dparser).

* Martin Nowak put these gists on the D newsgroup:
    - https://gist.github.com/1255439 - lexer generator
    - https://gist.github.com/1262321 - complete and fast D lexer


I should also add the parser from Dil and the one from SDC (At least,
I suppose there is one in SDC)
Jacob, do you have a link to your parser?

Also, Timon seems to have a parser, and JM Davis was working on a D lexer.
Jan 16 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 16 January 2013 at 10:52:39 UTC, Philippe Sigaud 
wrote:

 * Martin Nowak put these gists on the D newsgroup:
     - https://gist.github.com/1255439 - lexer generator
     - https://gist.github.com/1262321 - complete and fast D 
 lexer



 New link:

 https://github.com/dawgfoto/lexer



If it is going public, can we use a naming convention that make 
sense for tokens types ?
Jan 16 2013
prev sibling next sibling parent Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Wed, Jan 16, 2013 at 1:11 PM, Jacob Carlborg <doob me.com> wrote:

 That's a good start. We should put these on the wiki.



OK, I'll do that tonight. That will become a reflex ;)


 Jacob, do you have a link to your parser?




 I don't have a parser. That's why I'm complaining all the time that we
 should have a lexer and parser for D written in D :)



Drat.
Jan 16 2013
prev sibling parent Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Wed, Jan 16, 2013 at 8:46 PM, Jacob Carlborg <doob me.com> wrote:

 On 2013-01-16 17:41, Philippe Sigaud wrote:


 OK, I'll do that tonight. That will become a reflex ;)




 Great :)



It's already night-time were I live:

http://wiki.dlang.org/Lexers_Parsers

I added a link on the main page, in the Compilers & Tools subpart.
Jan 16 2013