www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Compiling DMD parser?

reply Markus Koskimies <markus reaaliaika.net> writes:
Hi,

I thought that I could play around a little bit with DMD parser. I have 
DMD 1.030 parser sources, and I made a simple makefile for just compiling 
those sources (with gcc 4.3.1).

---
Some questions:

- These are C++ sources, not C sources, although the suffix is .c and .h, 
right? A little bit sad, I thought that I could wrap the parser to D :( I 
have not much interest playing with C++ at freetime...

- Are these files meant to be compiled alone without making modifications 
to files? For example:

- In "attrib.c", if linux is defined, it tries to include mem.h from "../
root"; why not the one in the src/ directory?

- There is no such file that "id.h", which probably contains class/struct/
enum "Id"

- init.h does not include "hdrgen.h", while uses HdrGenState

---
How these parser codes are meant to be used? Am I missing something?
Jul 12 2008
next sibling parent reply Bill Baxter <dnewsgroup billbaxter.com> writes:
Markus Koskimies wrote:
 Hi,
 
 I thought that I could play around a little bit with DMD parser. I have 
 DMD 1.030 parser sources, and I made a simple makefile for just compiling 
 those sources (with gcc 4.3.1).
 
 ---
 Some questions:
 
 - These are C++ sources, not C sources, although the suffix is .c and .h, 
 right? A little bit sad, I thought that I could wrap the parser to D :( I 
 have not much interest playing with C++ at freetime...
 
 - Are these files meant to be compiled alone without making modifications 
 to files? For example:
 
 - In "attrib.c", if linux is defined, it tries to include mem.h from "../
 root"; why not the one in the src/ directory?
 
 - There is no such file that "id.h", which probably contains class/struct/
 enum "Id"
 
 - init.h does not include "hdrgen.h", while uses HdrGenState
 
 ---
 How these parser codes are meant to be used? Am I missing something?

They aren't. They're just meant to be for anyone who is interested in taking them and trying to build a compiler or other tool out of them. They don't compile into anything useful as-is. I think there's a project, DMDFE, based on those sources that is compileable though. http://www.dsource.org/projects/dmdfe --bb
Jul 12 2008
next sibling parent Markus Koskimies <markus reaaliaika.net> writes:
On Sun, 13 Jul 2008 13:58:27 +0900, Bill Baxter wrote:

 They aren't.  They're just meant to be for anyone who is interested in
 taking them and trying to build a compiler or other tool out of them.
 They don't compile into anything useful as-is.

Well, I thought that they could be compiled to object files, and you would be able to wrap & link them to your own project... But if they are meant just to be rewritten, that's of course different story.
 I think there's a project, DMDFE, based on those sources that is
 compileable though.  http://www.dsource.org/projects/dmdfe

I'll check it out.
Jul 12 2008
prev sibling next sibling parent reply Markus Koskimies <markus reaaliaika.net> writes:
On Sun, 13 Jul 2008 13:58:27 +0900, Bill Baxter wrote:

 I think there's a project, DMDFE, based on those sources that is
 compileable though.  http://www.dsource.org/projects/dmdfe

Yes, that worked, thanks! Next I'll try to figure out, how to easily import the parsed results to D... Hmmh, I might have an idea; if I modify that DMDFE to output the parse tree in some kind of ASCII stream, I might be able to construct it in D; hmmh, that sounds somewhat silly... Well, I try to figure out something. Again, thanks for that link! :D
Jul 12 2008
parent Bill Baxter <dnewsgroup billbaxter.com> writes:
Markus Koskimies wrote:
 On Sun, 13 Jul 2008 13:58:27 +0900, Bill Baxter wrote:
 
 I think there's a project, DMDFE, based on those sources that is
 compileable though.  http://www.dsource.org/projects/dmdfe

Yes, that worked, thanks! Next I'll try to figure out, how to easily import the parsed results to D... Hmmh, I might have an idea; if I modify that DMDFE to output the parse tree in some kind of ASCII stream, I might be able to construct it in D; hmmh, that sounds somewhat silly... Well, I try to figure out something. Again, thanks for that link! :D

No need to go ascii. Just wrap up the data in some structs and add some extern(C) functions that you can call from D which return those structs. Or if you're doing D2.0, then there's that extern(C++) thing that may work. --bb
Jul 12 2008
prev sibling parent Markus Koskimies <markus reaaliaika.net> writes:
On Sun, 13 Jul 2008 14:31:29 +0900, Bill Baxter wrote:

 Markus Koskimies wrote:
 On Sun, 13 Jul 2008 13:58:27 +0900, Bill Baxter wrote:
 
 I think there's a project, DMDFE, based on those sources that is
 compileable though.  http://www.dsource.org/projects/dmdfe

Yes, that worked, thanks! Next I'll try to figure out, how to easily import the parsed results to D... Hmmh, I might have an idea; if I modify that DMDFE to output the parse tree in some kind of ASCII stream, I might be able to construct it in D; hmmh, that sounds somewhat silly... Well, I try to figure out something. Again, thanks for that link! :D

No need to go ascii. Just wrap up the data in some structs and add some extern(C) functions that you can call from D which return those structs.

Yes, I thought that, too, but I thought that it could be quite an effort. Maybe later :) I thought that if I put the dmdfe dumping info from parsing, the same can be used for importing the tree to D. But I'll play around for awhile to find a suitable way of doing it.
 Or if you're doing D2.0, then there's that extern(C++) thing that may
 work.

Does it work with g++? I haven't yet used D2.0, but maybe it is time to try it out, too :)
Jul 12 2008
prev sibling next sibling parent reply Lars Ivar Igesund <larsivar igesund.net> writes:
Markus Koskimies wrote:

 Hi,
 
 I thought that I could play around a little bit with DMD parser. I have
 DMD 1.030 parser sources, and I made a simple makefile for just compiling
 those sources (with gcc 4.3.1).
 
 ---
 Some questions:
 
 - These are C++ sources, not C sources, although the suffix is .c and .h,
 right? A little bit sad, I thought that I could wrap the parser to D :( I
 have not much interest playing with C++ at freetime...
 
 - Are these files meant to be compiled alone without making modifications
 to files? For example:
 
 - In "attrib.c", if linux is defined, it tries to include mem.h from "../
 root"; why not the one in the src/ directory?
 
 - There is no such file that "id.h", which probably contains class/struct/
 enum "Id"
 
 - init.h does not include "hdrgen.h", while uses HdrGenState
 
 ---
 How these parser codes are meant to be used? Am I missing something?

The DParser project on dsource may be of interest too, it is a D port of the parsing parts of the DMD frontend. -- Lars Ivar Igesund blog at http://larsivi.net DSource, #d.tango & #D: larsivi Dancing the Tango
Jul 12 2008
parent Markus Koskimies <markus reaaliaika.net> writes:
On Sun, 13 Jul 2008 08:23:32 +0200, Lars Ivar Igesund wrote:

 The DParser project on dsource may be of interest too, it is a D port of
 the parsing parts of the DMD frontend.

Aargh, I have constantly mixed it with this one (this is found if you google it): http://dparser.sourceforge.net/ I have wondered that what it really has to do with D... But now I figured out, that using: http://www.dsource.org/projects/dparser It's exactly what I expected it to be :D
Jul 13 2008
prev sibling next sibling parent reply "Koroskin Denis" <2korden+dmd gmail.com> writes:
On Sun, 13 Jul 2008 08:33:54 +0400, Markus Koskimies  
<markus reaaliaika.net> wrote:

 Hi,

 I thought that I could play around a little bit with DMD parser. I have
 DMD 1.030 parser sources, and I made a simple makefile for just compiling
 those sources (with gcc 4.3.1).

 ---
 Some questions:

 - These are C++ sources, not C sources, although the suffix is .c and .h,
 right? A little bit sad, I thought that I could wrap the parser to D :( I
 have not much interest playing with C++ at freetime...

 - Are these files meant to be compiled alone without making modifications
 to files? For example:

 - In "attrib.c", if linux is defined, it tries to include mem.h from "../
 root"; why not the one in the src/ directory?

 - There is no such file that "id.h", which probably contains  
 class/struct/
 enum "Id"

 - init.h does not include "hdrgen.h", while uses HdrGenState

 ---
 How these parser codes are meant to be used? Am I missing something?

I'm working on something that could be of interest to you. I have a working DMD to D bridge via C abi. It's not complete but most important data structures are available from D, i.e. you can pass some .d files to it and get an AST back. I've written a small source-code analyzer as a proof-of-concept with it. Think of it as a DParser (http://www.dsource.org/projects/dparser) but with official implementation. A huge plus is that almost no maintaining effort needed. Bridge is written from scratch with a mix of C++ and D1 and is not based on DMDFE (http://www.dsource.org/projects/dmdfe). I haven't released the sources yet, so contact me if you would like to look at the them.
Jul 13 2008
parent reply Markus Koskimies <markus reaaliaika.net> writes:
On Mon, 14 Jul 2008 02:41:34 +0400, Koroskin Denis wrote:

 Think of it as a DParser (http://www.dsource.org/projects/dparser) but
 with official implementation. A huge plus is that almost no maintaining
 effort needed.

Yes, I was already wondering, how DParser will keep up-to-date. It would be a huge effort to go through the new C++ files and check, what need to be changed in DParser...
 Bridge is written from scratch with a mix of C++ and D1 and is not based
 on DMDFE (http://www.dsource.org/projects/dmdfe). I haven't released the
 sources yet, so contact me if you would like to look at the them.

I put some email :)
Jul 13 2008
next sibling parent Bill Baxter <dnewsgroup billbaxter.com> writes:
Markus Koskimies wrote:
 On Mon, 14 Jul 2008 02:41:34 +0400, Koroskin Denis wrote:
 
 Think of it as a DParser (http://www.dsource.org/projects/dparser) but
 with official implementation. A huge plus is that almost no maintaining
 effort needed.

Yes, I was already wondering, how DParser will keep up-to-date. It would be a huge effort to go through the new C++ files and check, what need to be changed in DParser...

Given that DParser is apparently still back at version 1.014, I think the answer to how it will keep up-to-date is simply "it won't". --bb
Jul 13 2008
prev sibling parent Leandro Lucarella <llucax gmail.com> writes:
Markus Koskimies, el 13 de julio a las 23:11 me escribiste:
 On Mon, 14 Jul 2008 02:41:34 +0400, Koroskin Denis wrote:
 
 Think of it as a DParser (http://www.dsource.org/projects/dparser) but
 with official implementation. A huge plus is that almost no maintaining
 effort needed.

Yes, I was already wondering, how DParser will keep up-to-date. It would be a huge effort to go through the new C++ files and check, what need to be changed in DParser...

It would be much more easy if the DMD official frontend were in a public repository like Phobos, and one could track it's development in a daily basis, not dealing with a huge contextless diff made from 2 released versions of DMD. Please, please, please walter, put the frontend in a public repository =) -- Leandro Lucarella (luca) | Blog colectivo: http://www.mazziblog.com.ar/blog/ ---------------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ---------------------------------------------------------------------------- Ya ni el cielo me quiere, ya ni la muerte me visita Ya ni el sol me calienta, ya ni el viento me acaricia
Jul 14 2008
prev sibling parent reply davidl <davidl 126.com> writes:
在 Sun, 13 Jul 2008 12:33:54 +0800,Markus Koskimies  
<markus reaaliaika.net> 写道:

 Hi,

 I thought that I could play around a little bit with DMD parser. I have
 DMD 1.030 parser sources, and I made a simple makefile for just compiling
 those sources (with gcc 4.3.1).

 ---
 Some questions:

 - These are C++ sources, not C sources, although the suffix is .c and .h,
 right? A little bit sad, I thought that I could wrap the parser to D :( I
 have not much interest playing with C++ at freetime...

 - Are these files meant to be compiled alone without making modifications
 to files? For example:

 - In "attrib.c", if linux is defined, it tries to include mem.h from "../
 root"; why not the one in the src/ directory?

 - There is no such file that "id.h", which probably contains  
 class/struct/
 enum "Id"

 - init.h does not include "hdrgen.h", while uses HdrGenState

 ---
 How these parser codes are meant to be used? Am I missing something?

actually dparser is not 1.014 anymore. it's somewhat 1.024ish the parsing ability has been improved a lot, it can parse itself, and some ctfes. 700 or so cases from dstress fail. -- 使用 Opera 革命性的电子邮件客户程序: http://www.opera.com/mail/
Jul 13 2008
next sibling parent Bill Baxter <dnewsgroup billbaxter.com> writes:
davidl wrote:
 在 Sun, 13 Jul 2008 12:33:54 +0800,Markus Koskimies 
 <markus reaaliaika.net> 写道:
 
 Hi,

 I thought that I could play around a little bit with DMD parser. I have
 DMD 1.030 parser sources, and I made a simple makefile for just compiling
 those sources (with gcc 4.3.1).

 ---
 Some questions:

 - These are C++ sources, not C sources, although the suffix is .c and .h,
 right? A little bit sad, I thought that I could wrap the parser to D :( I
 have not much interest playing with C++ at freetime...

 - Are these files meant to be compiled alone without making modifications
 to files? For example:

 - In "attrib.c", if linux is defined, it tries to include mem.h from "../
 root"; why not the one in the src/ directory?

 - There is no such file that "id.h", which probably contains 
 class/struct/
 enum "Id"

 - init.h does not include "hdrgen.h", while uses HdrGenState

 ---
 How these parser codes are meant to be used? Am I missing something?

actually dparser is not 1.014 anymore. it's somewhat 1.024ish the parsing ability has been improved a lot, it can parse itself, and some ctfes. 700 or so cases from dstress fail.

I hope you don't mind that I too the liberty of updating the dparser wiki page to reflect that information. --bb
Jul 13 2008
prev sibling next sibling parent reply Markus Koskimies <markus reaaliaika.net> writes:
On Mon, 14 Jul 2008 11:26:13 +0800, davidl wrote:

 actually dparser is not 1.014 anymore. it's somewhat 1.024ish the
 parsing ability has been improved a lot, it can parse itself, and some
 ctfes.
 700 or so cases from dstress fail.

:o I fetched the sources (I tried to select the latest ones), and attached that DParser to my own code. I copied the set-ups from trunk/d1.0/dparser/ dmd/main.d, and do parsing and semantic passes 1 to 3 to build the parsed tree. Problems: - Parser segfaults, if it parses std.stdio - I'm trying to write a tree walker; I have walk(X) functions for different types of elements (Module, FuncDeclaration, DeclStatement, ...). But I have hard times to determine the types of statements, since Statement class contains only a few "isXXXX()" functions. I tried to use typeid(typeof(s)), but it returns - as you may guess - always the type that was declared in the function arguments. --- I use DMD1.030, and I have those 1.030 phobos sources. I'll try to figure out how to extract that statement type information from parse tree, and could wait for newer versions of DParser. At the same time I'll try to get dil compiled; although dil does not generate code, it might still be a suitable platform to write a software to generate some common warnings (just for my personal use, I'm not intended to start to write "dlint"). Unfortunately it means that I need to figure out how those DSSS & Tango things work :)
Jul 13 2008
parent Leandro Lucarella <llucax gmail.com> writes:
Markus Koskimies, el 15 de julio a las 08:44 me escribiste:
[snip]
 And it reports:
 
 PASS 1: Building parse tree...
 semantic1 done
 semantic2 done
 semantic3 done 0
 PASS 2: Walking parse tree...
 UnusedLocal1.d(8): Unused variable: unused
 
 ...Just as it should do :D

Very nice!
 I'll put the full sources available somewhere (to my home pages, at 
 least), and continue developing walking & warning thing.

That would be very useful. -- Leandro Lucarella (luca) | Blog colectivo: http://www.mazziblog.com.ar/blog/ ---------------------------------------------------------------------------- GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145 104C 949E BFB6 5F5A 8D05) ---------------------------------------------------------------------------- careful to all animals (never washing spiders down the plughole), keep in contact with old friends (enjoy a drink now and then), will frequently check credit at (moral) bank (hole in the wall),
Jul 15 2008
prev sibling next sibling parent davidl <davidl 126.com> writes:
在 Mon, 14 Jul 2008 11:32:57 +0800,Bill Baxter  
<dnewsgroup billbaxter.com> 写道:

 davidl wrote:
 在 Sun, 13 Jul 2008 12:33:54 +0800,Markus Koskimies  
 <markus reaaliaika.net> 写道:

 Hi,

 I thought that I could play around a little bit with DMD parser. I have
 DMD 1.030 parser sources, and I made a simple makefile for just  
 compiling
 those sources (with gcc 4.3.1).

 ---
 Some questions:

 - These are C++ sources, not C sources, although the suffix is .c and  
 .h,
 right? A little bit sad, I thought that I could wrap the parser to D  
 :( I
 have not much interest playing with C++ at freetime...

 - Are these files meant to be compiled alone without making  
 modifications
 to files? For example:

 - In "attrib.c", if linux is defined, it tries to include mem.h from  
 "../
 root"; why not the one in the src/ directory?

 - There is no such file that "id.h", which probably contains  
 class/struct/
 enum "Id"

 - init.h does not include "hdrgen.h", while uses HdrGenState

 ---
 How these parser codes are meant to be used? Am I missing something?

the parsing ability has been improved a lot, it can parse itself, and some ctfes. 700 or so cases from dstress fail.

I hope you don't mind that I too the liberty of updating the dparser wiki page to reflect that information. --bb

Appreciated! Wish you get envolved! :D -- 使用 Opera 革命性的电子邮件客户程序: http://www.opera.com/mail/
Jul 13 2008
prev sibling next sibling parent davidl <davidl 126.com> writes:
在 Mon, 14 Jul 2008 12:07:00 +0800,Markus Koskimies  
<markus reaaliaika.net> 写道:

 On Mon, 14 Jul 2008 11:26:13 +0800, davidl wrote:

 actually dparser is not 1.014 anymore. it's somewhat 1.024ish the
 parsing ability has been improved a lot, it can parse itself, and some
 ctfes.
 700 or so cases from dstress fail.

:o I fetched the sources (I tried to select the latest ones), and attached that DParser to my own code. I copied the set-ups from trunk/d1.0/dparser/ dmd/main.d, and do parsing and semantic passes 1 to 3 to build the parsed tree. Problems: - Parser segfaults, if it parses std.stdio - I'm trying to write a tree walker; I have walk(X) functions for different types of elements (Module, FuncDeclaration, DeclStatement, ...). But I have hard times to determine the types of statements, since Statement class contains only a few "isXXXX()" functions. I tried to use typeid(typeof(s)), but it returns - as you may guess - always the type that was declared in the function arguments. --- I use DMD1.030, and I have those 1.030 phobos sources. I'll try to figure out how to extract that statement type information from parse tree, and could wait for newer versions of DParser. At the same time I'll try to get dil compiled; although dil does not generate code, it might still be a suitable platform to write a software to generate some common warnings (just for my personal use, I'm not intended to start to write "dlint"). Unfortunately it means that I need to figure out how those DSSS & Tango things work :)

err, i didn't maintain the trunk/d1.0/dparser/dmd/main.d and surely the d2.0 branch also not maintained at the moment. use d1.0/test_dparser It shouldn't segfault. std.stdio has been parsed a lot of times here :) It's a good start from there I think -- 使用 Opera 革命性的电子邮件客户程序: http://www.opera.com/mail/
Jul 13 2008
prev sibling next sibling parent Markus Koskimies <markus reaaliaika.net> writes:
On Mon, 14 Jul 2008 12:31:39 +0800, davidl wrote:

 err, i didn't maintain the trunk/d1.0/dparser/dmd/main.d and surely the
 d2.0 branch also not maintained at the moment.

 use d1.0/test_dparser

 It shouldn't segfault. std.stdio has been parsed a lot of times here :)
 It's a good start from there I think

Yes, that worked :) Sorry to use unmaintained things, but there is such a little information written in that package :) I just took first file I saw containing main, after I had written a somewhat functional Makefile :) But that does not solve my problem to determine, which kind of statement type there is in the tree; I can currently extract only those, which have isXXXX functions in the Statement class.
Jul 13 2008
prev sibling next sibling parent Markus Koskimies <markus reaaliaika.net> writes:
On Mon, 14 Jul 2008 05:45:59 +0000, Markus Koskimies wrote:

 But that does not solve my problem to determine, which kind of statement
 type there is in the tree; I can currently extract only those, which
 have isXXXX functions in the Statement class.

Here is an example: test.d: --- import std.stdio; void main() { int x; writefln("%d", x); return; } --- When I walk through the module (test), I get following result: --- test: module object: import std: import Func: void main() Function body: Compound statement: Compound statement: Decl statement: int x = x (in a declar Exp)= 0; ==> Unknown statement: (writefln)("%d",(x)); Compound statement: ==> Unknown statement: 0; Return statement: return ; --- I'm wondering how to extract the type of those statements... At least they are not any of the Statement.isXXXX -types.
Jul 13 2008
prev sibling next sibling parent Markus Koskimies <markus reaaliaika.net> writes:
On Mon, 14 Jul 2008 04:07:00 +0000, Markus Koskimies wrote:

 At the same time I'll try to get dil compiled; although dil does not
 generate code, it might still be a suitable platform to write a software
 to generate some common warnings (just for my personal use, I'm not
 intended to start to write "dlint"). Unfortunately it means that I need
 to figure out how those DSSS & Tango things work :)

I finally made it, but that was extremely hard :D Now dil compiles \o/ I'm not sure where to tell these findings, so I write them here; hopefully someone finds them with google, if he/she enters same problems... DSSS, GDC, Tango and Linux-AMD64: --- First thing to know is that currently GDC (nor DMD) does not work completely with 64-bit Linuxes. Everything else seems to work, but not vararg's (that is, writef & writefln). So, install a chroot'ed 32-bit environment, instructions: http://saftsack.fs.uni-bayreuth.de/~dun3/archives/using-a-x86-chroot-to- run-32-bit-apps-on-debian-amd64/94.html --- You may install DMD, it does not cause any harm :) --- Second thing; if you followed those instructions, Debian sid comes with newer GCC than is supported by GDC. So don't install it (it may work, but better to use correct version)! Install first GDC (currently GDC-4.1), and install the same version of GCC and G++ (4.1). Try to compile a simple hello.d with GDC to check, that it really works. With correct versions of all necessary things (gcc, g++, gdc and their libraries & utils), gdc should work OK. --- Next thing to install is DSSS. It should work also, quite easily. Make it use GDC (to be able to build tango with GDC). I was not able to install curl to chrooted sid, so I'm not able to use dsss net installs. --- When you have GDC and DSSS working, you can try to get tango working. I fought with tango for several hours. I don't really know what phase didn't work, but here are something: * I tried both dsss build/install, and build/install scripts in tango package (currently 0.99.6). Both seemed to work (no errors), but GDC didn't build my simple tango-hello.d; there were continuously error messages like: hello.o: In function `_Dmain': hello.d:(.text+0x20): undefined reference to `_D5tango2io7Console4CoutC5tango2io7Console7Console6Output' hello.d:(.text+0x2f): undefined reference to `_D5tango2io7Console7Console6Output6appendMFAaZC5tango2io7Console7Console6Output' hello.d:(.text+0x3b): undefined reference to `_D5tango2io7Console7Console6Output7newlineMFZC5tango2io7Console7Console6Output' hello.o:(.data+0x38): undefined reference to `_D5tango2io7Console12__ModuleInfoZ' There were probably two reasons for this: 1) libgphobos.a in /usr/lib/gcc/i486-linux-gnu/4.1.3/ was probably incorrectly build, and I didn't notice to update it. 2) It seems, that libDG-tango-core.a does not include all the necessary files. I ran "ar r libDG-tango-core.a *.o" in tango/ directory (where the libDG* files are), since there was object files like tango.core.Exception.o), which were not in that core library. --- * In tango directory, use script: ./build-tango.sh gdc ...To build libgtango.a, which you copy to correct location (good guess: / usr/lib/gcc/i486-linux-gnu/4.1.3/) * It could be easier first to try to get plain gdc builds to work: $ gdmd -version=Posix tango-hello.d -ofhello -W-lgtango --- Now when I got those things working, I'm not going to uninstall them and try again, so I can't tell step-by-step instructions for getting them working ;D
Jul 14 2008
prev sibling next sibling parent davidl <davidl 126.com> writes:
在 Mon, 14 Jul 2008 14:01:48 +0800,Markus Koskimies  
<markus reaaliaika.net> 写道:

 On Mon, 14 Jul 2008 05:45:59 +0000, Markus Koskimies wrote:

 But that does not solve my problem to determine, which kind of statement
 type there is in the tree; I can currently extract only those, which
 have isXXXX functions in the Statement class.

Here is an example: test.d: --- import std.stdio; void main() { int x; writefln("%d", x); return; } --- When I walk through the module (test), I get following result: --- test: module object: import std: import Func: void main() Function body: Compound statement: Compound statement: Decl statement: int x = x (in a declar Exp)= 0; ==> Unknown statement: (writefln)("%d",(x)); Compound statement: ==> Unknown statement: 0; Return statement: return ; --- I'm wondering how to extract the type of those statements... At least they are not any of the Statement.isXXXX -types.

Oh, it's pretty awesome you get that far in such a short time. I bet you can print instance.classinfo.name to get the type name of that instance. And I want to have your code be integrated into the dparser trunk for illustrating how to use dparser. -- 使用 Opera 革命性的电子邮件客户程序: http://www.opera.com/mail/
Jul 14 2008
prev sibling next sibling parent Markus Koskimies <markus reaaliaika.net> writes:
On Mon, 14 Jul 2008 18:59:16 +0800, davidl wrote:

 I'm wondering how to extract the type of those statements... At least
 they are not any of the Statement.isXXXX -types.

Oh, it's pretty awesome you get that far in such a short time. I bet you can print instance.classinfo.name to get the type name of that instance.

Hmmh, I'll try that... I'm still playing around to get used to these new D things (dsss, tango, tango-phobos -coexistence etc)...
 And I want to have your code be integrated into the dparser trunk for
 illustrating how to use dparser.

I needed to make some modifications to dparser sources to get them compiled to Linux (if I remeber correctly). I'll try to do that walking thing, and then I get fresh, unmodified dparser and see, if I get it compiled without any hacking.
Jul 14 2008
prev sibling next sibling parent Markus Koskimies <markus reaaliaika.net> writes:
On Mon, 14 Jul 2008 04:07:00 +0000, Markus Koskimies wrote:

 actually dparser is not 1.014 anymore. it's somewhat 1.024ish the
 parsing ability has been improved a lot, it can parse itself, and some
 ctfes.
 700 or so cases from dstress fail.

:o I fetched the sources (I tried to select the latest ones), and attached that DParser to my own code.

Jihuu! Now it works! I wrote a somewhat general purpose AST walker. It can be inherited and modified for different purposes. As an example, I made a class for scanning unused local variables: class UnusedScan : ASTWalk { /* To track down variable usage from nested functions, the * variable reference table is global. */ bool[VarDeclaration] referenced; /* Frame holds list of local variable definitions */ class Frame { VarDeclaration[] vars; } Stack!(Frame) frames; //------------------------------------------------------------------------- // Whenever we meet a new variable frame, push it. When popping it out, // we check, if the variable was referenced. //------------------------------------------------------------------------- void push() { frames.push( new Frame() ); } void pop() { Frame frame = frames.pop(); foreach(VarDeclaration var; frame.vars) { if(referenced[var]) continue; writefln("%s: Unused variable: %s", var.loc.toString(), var.toString() ); } } //------------------------------------------------------------------------- // Constructor //------------------------------------------------------------------------- this() { frames = new Stack!(Frame)(); } //------------------------------------------------------------------------- // Whenever we meet a function, class or similar, make a new frame. // After scanning, check if there were unreferenced variables. //------------------------------------------------------------------------- override void walk(Module m) { push(); super.walk(m); pop(); } override void walk(ClassDeclaration d) { push(); super.walk(d); pop(); } override void walk(FuncDeclaration f) { push(); super.walk(f); pop(); } //------------------------------------------------------------------------- // For every var declaration, put it to frame. //------------------------------------------------------------------------- override void walk(VarDeclaration var) { referenced[var] = false; frames.top().vars ~= var; } //------------------------------------------------------------------------- // When var is used, mark it to frame. //------------------------------------------------------------------------- override void walk(VarExp var) { referenced[cast(VarDeclaration)var.var] = true; } } --- Then I tested it against a small D program: import std.stdio; void main() { int a, b; int unused; // a is used inside expression writefln("%d", 23+a); // b is used from local function int f1() { int f2() { return b; } return f2(); } // unused is shadowed; still, the main unused should remain // not referred. void f3() { int unused; unused = 1; } return; } --- And it reports: PASS 1: Building parse tree... semantic1 done semantic2 done semantic3 done 0 PASS 2: Walking parse tree... UnusedLocal1.d(8): Unused variable: unused ...Just as it should do :D I'll put the full sources available somewhere (to my home pages, at least), and continue developing walking & warning thing. Davidl, do you want the sources for your repository?
Jul 15 2008
prev sibling next sibling parent Markus Koskimies <markus reaaliaika.net> writes:
On Tue, 15 Jul 2008 10:29:01 -0300, Leandro Lucarella wrote:

 I'll put the full sources available somewhere (to my home pages, at
 least), and continue developing walking & warning thing.

That would be very useful.

I'm currently hacking more scanners: - Scanning/warning, if function return value is discarded, i.e. int f() { ... } f(); // Warn cast(void)f(); // Don't warn - Warning for suspicious looking infinite loops, i.e. for(;3.1415926535894;) { ... } // Warn for(;;) { } // Don't warn There are some problems to make them correct, since DParser does not always store all necessary information, e.g. "for(;true;)" = "for(;;)" I probably include scanning of constant if/switch values with the loop scanning... That general purpose walking class makes implementation of these kinds of checkings relatively easy. There will surely missing / wrongly working pieces in the code, but those can be corrected later :)
Jul 15 2008
prev sibling next sibling parent davidl <davidl 126.com> writes:
在 Tue, 15 Jul 2008 16:44:25 +0800,Markus Koskimies  
<markus reaaliaika.net> 写道:

 On Mon, 14 Jul 2008 04:07:00 +0000, Markus Koskimies wrote:

 actually dparser is not 1.014 anymore. it's somewhat 1.024ish the
 parsing ability has been improved a lot, it can parse itself, and some
 ctfes.
 700 or so cases from dstress fail.

:o I fetched the sources (I tried to select the latest ones), and attached that DParser to my own code.

Jihuu! Now it works! I wrote a somewhat general purpose AST walker. It can be inherited and modified for different purposes. As an example, I made a class for scanning unused local variables: class UnusedScan : ASTWalk { /* To track down variable usage from nested functions, the * variable reference table is global. */ bool[VarDeclaration] referenced; /* Frame holds list of local variable definitions */ class Frame { VarDeclaration[] vars; } Stack!(Frame) frames; //------------------------------------------------------------------------- // Whenever we meet a new variable frame, push it. When popping it out, // we check, if the variable was referenced. //------------------------------------------------------------------------- void push() { frames.push( new Frame() ); } void pop() { Frame frame = frames.pop(); foreach(VarDeclaration var; frame.vars) { if(referenced[var]) continue; writefln("%s: Unused variable: %s", var.loc.toString(), var.toString() ); } } //------------------------------------------------------------------------- // Constructor //------------------------------------------------------------------------- this() { frames = new Stack!(Frame)(); } //------------------------------------------------------------------------- // Whenever we meet a function, class or similar, make a new frame. // After scanning, check if there were unreferenced variables. //------------------------------------------------------------------------- override void walk(Module m) { push(); super.walk(m); pop(); } override void walk(ClassDeclaration d) { push(); super.walk(d); pop(); } override void walk(FuncDeclaration f) { push(); super.walk(f); pop(); } //------------------------------------------------------------------------- // For every var declaration, put it to frame. //------------------------------------------------------------------------- override void walk(VarDeclaration var) { referenced[var] = false; frames.top().vars ~= var; } //------------------------------------------------------------------------- // When var is used, mark it to frame. //------------------------------------------------------------------------- override void walk(VarExp var) { referenced[cast(VarDeclaration)var.var] = true; } } --- Then I tested it against a small D program: import std.stdio; void main() { int a, b; int unused; // a is used inside expression writefln("%d", 23+a); // b is used from local function int f1() { int f2() { return b; } return f2(); } // unused is shadowed; still, the main unused should remain // not referred. void f3() { int unused; unused = 1; } return; } --- And it reports: PASS 1: Building parse tree... semantic1 done semantic2 done semantic3 done 0 PASS 2: Walking parse tree... UnusedLocal1.d(8): Unused variable: unused ...Just as it should do :D I'll put the full sources available somewhere (to my home pages, at least), and continue developing walking & warning thing. Davidl, do you want the sources for your repository?

Of course! Send me the patch? If you get a dsource account, maybe you're the right guy to cooperate on this interesting project, I may grant you write access ;) -- 使用 Opera 革命性的电子邮件客户程序: http://www.opera.com/mail/
Jul 15 2008
prev sibling next sibling parent davidl <davidl 126.com> writes:
在 Tue, 15 Jul 2008 22:37:56 +0800,Markus Koskimies  
<markus reaaliaika.net> 写道:

 On Tue, 15 Jul 2008 10:29:01 -0300, Leandro Lucarella wrote:

 I'll put the full sources available somewhere (to my home pages, at
 least), and continue developing walking & warning thing.

That would be very useful.

I'm currently hacking more scanners: - Scanning/warning, if function return value is discarded, i.e. int f() { ... } f(); // Warn cast(void)f(); // Don't warn - Warning for suspicious looking infinite loops, i.e. for(;3.1415926535894;) { ... } // Warn for(;;) { } // Don't warn There are some problems to make them correct, since DParser does not always store all necessary information, e.g. "for(;true;)" = "for(;;)" I probably include scanning of constant if/switch values with the loop scanning... That general purpose walking class makes implementation of these kinds of checkings relatively easy. There will surely missing / wrongly working pieces in the code, but those can be corrected later :)

The unused variable bug has been appeared in the bugzilla, and yes, i did think about it a bit. The way I think is modify the VarDeclaration to have a member flag to show this var is used or not. But this require a more complicated way of coding, your walker might be the better way to code. If we can balance the performance issue, I think we open something really interesting in balancing of design pattern and performance. -- 使用 Opera 革命性的电子邮件客户程序: http://www.opera.com/mail/
Jul 15 2008
prev sibling next sibling parent Markus Koskimies <markus reaaliaika.net> writes:
On Tue, 15 Jul 2008 23:00:07 +0800, davidl wrote:

 Of course!
 Send me the patch? If you get a dsource account, maybe you're the right
 guy to cooperate on this interesting project, I may grant you write
 access ;)

I'll put you some email, just wait-a-moment :)
Jul 15 2008
prev sibling next sibling parent Markus Koskimies <markus reaaliaika.net> writes:
On Tue, 15 Jul 2008 08:44:25 +0000, Markus Koskimies wrote:

 I'll put the full sources available somewhere (to my home pages, at
 least), and continue developing walking & warning thing.

Sources are available: http://reaaliaika.net/showarea.php/download/dwalk dwalk.tar.ge opens to directory named dwalk. dwalk/dparser should contain link to DParser sources (you may also upload them to dwalk/ directory with svn etc). Package contains: - Sources: dwalk/src/ - Test files: dwalk/testfiles About sources: You probably first take a look to main.d. Then you may want to look at the scanners (ScanXXXModule.d). Finally, you might want to take a look at the ASTWalk (ASTWalking.d). If you get the thing compiled, you should have executable named dwalk, which can: - Detect unused things; there may still be some false reporting, as well as it can surely undetect some unused things. imports are not checked at all. Nor templates (or they may be, if they go with the defaults). - Detect expression result discarding, i.e. unused return values. There are some pecularities with "void main(...)" functions, which cause false alarms. - Detect constant conditions, in for/while/do and if/switch; take a look at the testfiles/TestInfLoop1.d for seeing, which kind of infinite or dead loops are scanned. Any reports, comments & suggestions are welcome :D
Jul 15 2008
prev sibling parent Markus Koskimies <markus reaaliaika.net> writes:
On Tue, 15 Jul 2008 20:40:21 +0000, Markus Koskimies wrote:

 On Tue, 15 Jul 2008 08:44:25 +0000, Markus Koskimies wrote:
 
 I'll put the full sources available somewhere (to my home pages, at
 least), and continue developing walking & warning thing.

Sources are available: http://reaaliaika.net/showarea.php/download/dwalk

Oh, some additions... Packet includes binary for i386-linux. Packet compiles with DMD1.0 and phobos. If compiling to linux, you may need to hack (current) DParser sources a little; if I remember correctly, change memicmp -> memcmp somewhere (changes things case sensitive, bu so what), and somewhere else there was function like DebugOutputA, change it to writefln.
Jul 15 2008