www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.ide - Future of Descent and D Eclipse IDE

reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
Hi.

I'm now in a position where I can dedicate a lot of free time for an 
open source project, and I'm seriously considering going back working on 
the D Eclipse IDE project. I worked on Mmrnmhrm a couple of years ago, 
as part of my thesis, which led to some restrictions on the kinds of 
tasks I should work on. Now I don't have that issue, I have (almost) 
complete freedom on what I can work on, and in particular I would like 
to unify the two current efforts for a D Eclipse IDE: Descent and 
Mmrnmhrm, as there is a lot of work being put into both (especially 
Descent ^^ )

Now, Ary has been inactive for quite a while, and he said he wasn't 
interested in working in Descent any more :( . This means I'm currently 
the only Eclipse developer, so I could just do things in the way I'm 
thinking of, but Ary, I would still like to get your input, because I 
hope one day you might be interested in coming back to Descent 
development. :)

There a few issues and ideas I would like to establish for the future D 
Eclipse IDE project. But for now I'll just mention one which I think is 
quite important:

* I think the DMD frontend Java port should be separated into its own 
plugin/bundle, and not be part of the descent.core plugin. The first 
reason is for abstraction and modularization: This project is quite 
distinct on its own, and separation would make it a bit easier for 
people to understand it, and work on it alone (especially since it would 
require almost no knowledge about Eclipse), or for it to be used in 
other projects (like it is now for Mmrnmhrm). Also this allows the 
plugin to not be a singleton, so there could be several plugin instances 
with different versions running at the same time.

Separation purely for modularization should be reason enough, but if you 
are not convinced, there is a more serious issue which regards licenses: 
descent.core source is (heavily) based on the JDT core, which means this 
derived code must be licensed under the EPL. Same for the other Descent 
plugins. But the DMD frontend is licensed under the Artistic License. So 
unless the frontend port could be relicensed as EPL as well, then it 
should be in its own plugin. This is so that each plugin has its own 
license which applies to all its underlying code, to make things clearer.

Ary, I would like to know if by any chance you are interested in doing 
that separation in the near future. If not, I can do it myself, but it 
might take a bit longer.


Cheers

-- 
Bruno Medeiros - Software Engineer
May 24 2010
next sibling parent reply Trass3r <un known.com> writes:
Hope you can update Descent! :)
I really like it.
May 24 2010
parent Jacob Carlborg <doob me.com> writes:
On 2010-05-25 01.00, Trass3r wrote:
 Hope you can update Descent! :)
 I really like it.

Me too. -- /Jacob Carlborg
May 25 2010
prev sibling parent reply Ary Borenszweig <ary esperanto.org.ar> writes:
On 05/24/2010 08:38 AM, Bruno Medeiros wrote:
 Hi.

 I'm now in a position where I can dedicate a lot of free time for an
 open source project, and I'm seriously considering going back working on
 the D Eclipse IDE project. I worked on Mmrnmhrm a couple of years ago,
 as part of my thesis, which led to some restrictions on the kinds of
 tasks I should work on. Now I don't have that issue, I have (almost)
 complete freedom on what I can work on, and in particular I would like
 to unify the two current efforts for a D Eclipse IDE: Descent and
 Mmrnmhrm, as there is a lot of work being put into both (especially
 Descent ^^ )

 Now, Ary has been inactive for quite a while, and he said he wasn't
 interested in working in Descent any more :(

I know I told you that, but now that I think of it, it's not that I'm not interested. The project has grown too big and in the last releases I added nice features without thinking much about the design and the flexibility of growth... so now I feel the project is kind of a mess and it's very hard to continue it. The problems I see are: * Porting DMD source to Java was done manually and it's a very boring and long task, and we need to find a way to automate it if we'd like to support really good integration with the language (I mean, real semantic value, and because D is not dynamic I think this is worth it). * The DMD source was modified a little bit for performance reasons and for integration with Descent so now it contains bugs that are very hard to fix. * The code is just too big because it has a lot of code from JDT, modified a little bit, and that makes it hard for other developers to join. . This means I'm currently
 the only Eclipse developer, so I could just do things in the way I'm
 thinking of, but Ary, I would still like to get your input, because I
 hope one day you might be interested in coming back to Descent
 development. :)

I can program now and then if I have time, but what I'd really like is to plan how to start things almost from scratch and think of a plan of doing it well. Maybe Descent could be done with IMP, I don't think using JDT's source code will be good for the project. The only problem I see with IMP (or DLTK) is that they don't support some of the features Descent already provides... but as IMP gets better Descent will automatically get those updates.
 There a few issues and ideas I would like to establish for the future D
 Eclipse IDE project. But for now I'll just mention one which I think is
 quite important:

 * I think the DMD frontend Java port should be separated into its own
 plugin/bundle, and not be part of the descent.core plugin. The first
 reason is for abstraction and modularization: This project is quite
 distinct on its own, and separation would make it a bit easier for
 people to understand it, and work on it alone (especially since it would
 require almost no knowledge about Eclipse), or for it to be used in
 other projects (like it is now for Mmrnmhrm). Also this allows the
 plugin to not be a singleton, so there could be several plugin instances
 with different versions running at the same time.

Yes, I would like that! And I think it's very necessary. Then you can plug this module into any IDE implementation you want.
 Separation purely for modularization should be reason enough, but if you
 are not convinced, there is a more serious issue which regards licenses:
 descent.core source is (heavily) based on the JDT core, which means this
 derived code must be licensed under the EPL. Same for the other Descent
 plugins. But the DMD frontend is licensed under the Artistic License. So
 unless the frontend port could be relicensed as EPL as well, then it
 should be in its own plugin. This is so that each plugin has its own
 license which applies to all its underlying code, to make things clearer.

Ah, um, right. :-P
 Ary, I would like to know if by any chance you are interested in doing
 that separation in the near future. If not, I can do it myself, but it
 might take a bit longer.

Yes, I would like to do it, but first I think we need to think if we want to a) keep using the Java port and little by little upgrade it to the latest dmd versions, b) start another port but build a tool to more or less automate that process (Frank Benoit started something in that regard but I don't know what happened then) or c) make a less semantic-aware IDE like Mmrnmhrm (the good thing is that it might be simpler and faster, but the bad thing is that we might not get all the features we'd want). Thanks for bringing this topic! A lot of effort has been put in Descent and it would maybe be sad if the project is abandoned... (but I really feel I'm stuck now and I don't know how to advance... or at least I know I would have to keep porting C++ code to Java, but... http://www.explosm.net/comics/1804/ ) Cheers!
May 24 2010
next sibling parent dennis luehring <dl.soluz gmx.net> writes:
Am 25.05.2010 04:15, schrieb Ary Borenszweig:
 I can program now and then if I have time, but what I'd really like is
 to plan how to start things almost from scratch and think of a plan of
 doing it well.

you should try to bring some of your needed features directly into the official compiler suite - like the (far too "small" for your needs) json output - maybe there is a way to integrate your needs without stressing the code base (and walter) too much
May 24 2010
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2010-05-25 04.15, Ary Borenszweig wrote:
 On 05/24/2010 08:38 AM, Bruno Medeiros wrote:
 Hi.

 I'm now in a position where I can dedicate a lot of free time for an
 open source project, and I'm seriously considering going back working on
 the D Eclipse IDE project. I worked on Mmrnmhrm a couple of years ago,
 as part of my thesis, which led to some restrictions on the kinds of
 tasks I should work on. Now I don't have that issue, I have (almost)
 complete freedom on what I can work on, and in particular I would like
 to unify the two current efforts for a D Eclipse IDE: Descent and
 Mmrnmhrm, as there is a lot of work being put into both (especially
 Descent ^^ )

 Now, Ary has been inactive for quite a while, and he said he wasn't
 interested in working in Descent any more :(

I know I told you that, but now that I think of it, it's not that I'm not interested. The project has grown too big and in the last releases I added nice features without thinking much about the design and the flexibility of growth... so now I feel the project is kind of a mess and it's very hard to continue it. The problems I see are: * Porting DMD source to Java was done manually and it's a very boring and long task, and we need to find a way to automate it if we'd like to support really good integration with the language (I mean, real semantic value, and because D is not dynamic I think this is worth it).

I have two suggestions for this problem: 1. Could DMD be compiled to a dynamic library and then be used like a plugin, using JNI to interact between the compiler and the plugin. 2. Another suggestion is to use http://doc.trolltech.com/qtjambi-4.4/html/com/trolltech/qt/qt ambi-generator.html or http://www.gccxml.org/HTML/Index.html to either port DMD or create interfaces to it and then use it as the first suggestion.
 * The DMD source was modified a little bit for performance reasons and
 for integration with Descent so now it contains bugs that are very hard
 to fix.
 * The code is just too big because it has a lot of code from JDT,
 modified a little bit, and that makes it hard for other developers to join.
 . This means I'm currently
 the only Eclipse developer, so I could just do things in the way I'm
 thinking of, but Ary, I would still like to get your input, because I
 hope one day you might be interested in coming back to Descent
 development. :)

I can program now and then if I have time, but what I'd really like is to plan how to start things almost from scratch and think of a plan of doing it well. Maybe Descent could be done with IMP, I don't think using JDT's source code will be good for the project. The only problem I see with IMP (or DLTK) is that they don't support some of the features Descent already provides... but as IMP gets better Descent will automatically get those updates.
 There a few issues and ideas I would like to establish for the future D
 Eclipse IDE project. But for now I'll just mention one which I think is
 quite important:

 * I think the DMD frontend Java port should be separated into its own
 plugin/bundle, and not be part of the descent.core plugin. The first
 reason is for abstraction and modularization: This project is quite
 distinct on its own, and separation would make it a bit easier for
 people to understand it, and work on it alone (especially since it would
 require almost no knowledge about Eclipse), or for it to be used in
 other projects (like it is now for Mmrnmhrm). Also this allows the
 plugin to not be a singleton, so there could be several plugin instances
 with different versions running at the same time.

Yes, I would like that! And I think it's very necessary. Then you can plug this module into any IDE implementation you want.
 Separation purely for modularization should be reason enough, but if you
 are not convinced, there is a more serious issue which regards licenses:
 descent.core source is (heavily) based on the JDT core, which means this
 derived code must be licensed under the EPL. Same for the other Descent
 plugins. But the DMD frontend is licensed under the Artistic License. So
 unless the frontend port could be relicensed as EPL as well, then it
 should be in its own plugin. This is so that each plugin has its own
 license which applies to all its underlying code, to make things clearer.

Ah, um, right. :-P
 Ary, I would like to know if by any chance you are interested in doing
 that separation in the near future. If not, I can do it myself, but it
 might take a bit longer.

Yes, I would like to do it, but first I think we need to think if we want to a) keep using the Java port and little by little upgrade it to the latest dmd versions, b) start another port but build a tool to more or less automate that process (Frank Benoit started something in that regard but I don't know what happened then) or c) make a less semantic-aware IDE like Mmrnmhrm (the good thing is that it might be simpler and faster, but the bad thing is that we might not get all the features we'd want).

If you refer to the project Tioport by Frank Benoit, it was a project to automate porting Java code to D, I don't see how it would help in this case.
 Thanks for bringing this topic! A lot of effort has been put in Descent
 and it would maybe be sad if the project is abandoned... (but I really
 feel I'm stuck now and I don't know how to advance... or at least I know
 I would have to keep porting C++ code to Java, but...

 http://www.explosm.net/comics/1804/

 )

 Cheers!

-- /Jacob Carlborg
May 25 2010
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 25/05/2010 14:06, Jacob Carlborg wrote:
 On 2010-05-25 04.15, Ary Borenszweig wrote:
 On 05/24/2010 08:38 AM, Bruno Medeiros wrote:
 Hi.

 I'm now in a position where I can dedicate a lot of free time for an
 open source project, and I'm seriously considering going back working on
 the D Eclipse IDE project. I worked on Mmrnmhrm a couple of years ago,
 as part of my thesis, which led to some restrictions on the kinds of
 tasks I should work on. Now I don't have that issue, I have (almost)
 complete freedom on what I can work on, and in particular I would like
 to unify the two current efforts for a D Eclipse IDE: Descent and
 Mmrnmhrm, as there is a lot of work being put into both (especially
 Descent ^^ )

 Now, Ary has been inactive for quite a while, and he said he wasn't
 interested in working in Descent any more :(

I know I told you that, but now that I think of it, it's not that I'm not interested. The project has grown too big and in the last releases I added nice features without thinking much about the design and the flexibility of growth... so now I feel the project is kind of a mess and it's very hard to continue it. The problems I see are: * Porting DMD source to Java was done manually and it's a very boring and long task, and we need to find a way to automate it if we'd like to support really good integration with the language (I mean, real semantic value, and because D is not dynamic I think this is worth it).

I have two suggestions for this problem: 1. Could DMD be compiled to a dynamic library and then be used like a plugin, using JNI to interact between the compiler and the plugin.

Nope, the compiler generates a big structure, an AST, which is composed of nodes from a complex hierarchy of classes. Transferring such sctructure across JNI would be incredibly hard do implement, not to mention probably inefficient. JNI (as with most any C interfacing) is good mostly just for calling C methods with simple parameters. The other option would be for the compiler to expose just a thin API (without big data structures), and have the IDE query the semantic functionality directly to the compiler. But then the frontend itself would have to be extended a lot, which would me a lot of complicated coding in C... argh, no way... -- Bruno Medeiros - Software Engineer
May 25 2010
parent reply Lutger <lutger.blijdestijn gmail.com> writes:
Bruno Medeiros wrote:

 On 25/05/2010 14:06, Jacob Carlborg wrote:
 On 2010-05-25 04.15, Ary Borenszweig wrote:
 On 05/24/2010 08:38 AM, Bruno Medeiros wrote:
 Hi.

 I'm now in a position where I can dedicate a lot of free time for an
 open source project, and I'm seriously considering going back working
 on the D Eclipse IDE project. I worked on Mmrnmhrm a couple of years
 ago, as part of my thesis, which led to some restrictions on the kinds
 of tasks I should work on. Now I don't have that issue, I have (almost)
 complete freedom on what I can work on, and in particular I would like
 to unify the two current efforts for a D Eclipse IDE: Descent and
 Mmrnmhrm, as there is a lot of work being put into both (especially
 Descent ^^ )

 Now, Ary has been inactive for quite a while, and he said he wasn't
 interested in working in Descent any more :(

I know I told you that, but now that I think of it, it's not that I'm not interested. The project has grown too big and in the last releases I added nice features without thinking much about the design and the flexibility of growth... so now I feel the project is kind of a mess and it's very hard to continue it. The problems I see are: * Porting DMD source to Java was done manually and it's a very boring and long task, and we need to find a way to automate it if we'd like to support really good integration with the language (I mean, real semantic value, and because D is not dynamic I think this is worth it).

I have two suggestions for this problem: 1. Could DMD be compiled to a dynamic library and then be used like a plugin, using JNI to interact between the compiler and the plugin.

Nope, the compiler generates a big structure, an AST, which is composed of nodes from a complex hierarchy of classes. Transferring such sctructure across JNI would be incredibly hard do implement, not to mention probably inefficient. JNI (as with most any C interfacing) is good mostly just for calling C methods with simple parameters. The other option would be for the compiler to expose just a thin API (without big data structures), and have the IDE query the semantic functionality directly to the compiler. But then the frontend itself would have to be extended a lot, which would me a lot of complicated coding in C... argh, no way...

Is working with ddmd an option?
Jun 05 2010
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 05/06/2010 09:17, Lutger wrote:
 Bruno Medeiros wrote:

 On 25/05/2010 14:06, Jacob Carlborg wrote:
 On 2010-05-25 04.15, Ary Borenszweig wrote:
 On 05/24/2010 08:38 AM, Bruno Medeiros wrote:
 Hi.

 I'm now in a position where I can dedicate a lot of free time for an
 open source project, and I'm seriously considering going back working
 on the D Eclipse IDE project. I worked on Mmrnmhrm a couple of years
 ago, as part of my thesis, which led to some restrictions on the kinds
 of tasks I should work on. Now I don't have that issue, I have (almost)
 complete freedom on what I can work on, and in particular I would like
 to unify the two current efforts for a D Eclipse IDE: Descent and
 Mmrnmhrm, as there is a lot of work being put into both (especially
 Descent ^^ )

 Now, Ary has been inactive for quite a while, and he said he wasn't
 interested in working in Descent any more :(

I know I told you that, but now that I think of it, it's not that I'm not interested. The project has grown too big and in the last releases I added nice features without thinking much about the design and the flexibility of growth... so now I feel the project is kind of a mess and it's very hard to continue it. The problems I see are: * Porting DMD source to Java was done manually and it's a very boring and long task, and we need to find a way to automate it if we'd like to support really good integration with the language (I mean, real semantic value, and because D is not dynamic I think this is worth it).

I have two suggestions for this problem: 1. Could DMD be compiled to a dynamic library and then be used like a plugin, using JNI to interact between the compiler and the plugin.

Nope, the compiler generates a big structure, an AST, which is composed of nodes from a complex hierarchy of classes. Transferring such sctructure across JNI would be incredibly hard do implement, not to mention probably inefficient. JNI (as with most any C interfacing) is good mostly just for calling C methods with simple parameters. The other option would be for the compiler to expose just a thin API (without big data structures), and have the IDE query the semantic functionality directly to the compiler. But then the frontend itself would have to be extended a lot, which would me a lot of complicated coding in C... argh, no way...

Is working with ddmd an option?

Nah. Since DDMD is a direct port of DMD, the situation would not be much different: you would have a lot of C-like code in D, as well as a Walter-like design :P, and you wouldn't be able to change either of those things, because then you would make it significantly hard (if not impractical) to update the DDMD port with new DMD changes. That's exactly the same problem there is with the Java port of DMD. I'm now pretty sure the only good long-term strategy is to have a compiler/engine designed from scratch to work with IDE functionality. Although I didnt realize until now, thanks to your comment, that it might be feasible for this engine to be separate, (either as a process, or as library), from the Java IDE, and thus the engine could be in a different language than Java. Hum, this is quite interesting, one very big reason being that then it could be reused by different D IDEs, or other sorts of projects... -- Bruno Medeiros - Software Engineer
Jun 15 2010
parent reply Lutger <lutger.blijdestijn gmail.com> writes:
Bruno Medeiros wrote:

 On 05/06/2010 09:17, Lutger wrote:
 Bruno Medeiros wrote:

 On 25/05/2010 14:06, Jacob Carlborg wrote:
 On 2010-05-25 04.15, Ary Borenszweig wrote:
 On 05/24/2010 08:38 AM, Bruno Medeiros wrote:
 Hi.

 I'm now in a position where I can dedicate a lot of free time for an
 open source project, and I'm seriously considering going back working
 on the D Eclipse IDE project. I worked on Mmrnmhrm a couple of years
 ago, as part of my thesis, which led to some restrictions on the
 kinds of tasks I should work on. Now I don't have that issue, I have
 (almost) complete freedom on what I can work on, and in particular I
 would like to unify the two current efforts for a D Eclipse IDE:
 Descent and Mmrnmhrm, as there is a lot of work being put into both
 (especially Descent ^^ )

 Now, Ary has been inactive for quite a while, and he said he wasn't
 interested in working in Descent any more :(

I know I told you that, but now that I think of it, it's not that I'm not interested. The project has grown too big and in the last releases I added nice features without thinking much about the design and the flexibility of growth... so now I feel the project is kind of a mess and it's very hard to continue it. The problems I see are: * Porting DMD source to Java was done manually and it's a very boring and long task, and we need to find a way to automate it if we'd like to support really good integration with the language (I mean, real semantic value, and because D is not dynamic I think this is worth it).

I have two suggestions for this problem: 1. Could DMD be compiled to a dynamic library and then be used like a plugin, using JNI to interact between the compiler and the plugin.

Nope, the compiler generates a big structure, an AST, which is composed of nodes from a complex hierarchy of classes. Transferring such sctructure across JNI would be incredibly hard do implement, not to mention probably inefficient. JNI (as with most any C interfacing) is good mostly just for calling C methods with simple parameters. The other option would be for the compiler to expose just a thin API (without big data structures), and have the IDE query the semantic functionality directly to the compiler. But then the frontend itself would have to be extended a lot, which would me a lot of complicated coding in C... argh, no way...

Is working with ddmd an option?

Nah. Since DDMD is a direct port of DMD, the situation would not be much different: you would have a lot of C-like code in D, as well as a Walter-like design :P, and you wouldn't be able to change either of those things, because then you would make it significantly hard (if not impractical) to update the DDMD port with new DMD changes. That's exactly the same problem there is with the Java port of DMD. I'm now pretty sure the only good long-term strategy is to have a compiler/engine designed from scratch to work with IDE functionality. Although I didnt realize until now, thanks to your comment, that it might be feasible for this engine to be separate, (either as a process, or as library), from the Java IDE, and thus the engine could be in a different language than Java. Hum, this is quite interesting, one very big reason being that then it could be reused by different D IDEs, or other sorts of projects...

That, and being written in D, would make it more attractive. There is another option with a headstart: dil, if gpl is not a problem.
Jun 15 2010
parent Jacob Carlborg <doob me.com> writes:
On 2010-06-15 21:37, Lutger wrote:
 Bruno Medeiros wrote:

 On 05/06/2010 09:17, Lutger wrote:
 Bruno Medeiros wrote:

 On 25/05/2010 14:06, Jacob Carlborg wrote:
 On 2010-05-25 04.15, Ary Borenszweig wrote:
 On 05/24/2010 08:38 AM, Bruno Medeiros wrote:
 Hi.

 I'm now in a position where I can dedicate a lot of free time for an
 open source project, and I'm seriously considering going back working
 on the D Eclipse IDE project. I worked on Mmrnmhrm a couple of years
 ago, as part of my thesis, which led to some restrictions on the
 kinds of tasks I should work on. Now I don't have that issue, I have
 (almost) complete freedom on what I can work on, and in particular I
 would like to unify the two current efforts for a D Eclipse IDE:
 Descent and Mmrnmhrm, as there is a lot of work being put into both
 (especially Descent ^^ )

 Now, Ary has been inactive for quite a while, and he said he wasn't
 interested in working in Descent any more :(

I know I told you that, but now that I think of it, it's not that I'm not interested. The project has grown too big and in the last releases I added nice features without thinking much about the design and the flexibility of growth... so now I feel the project is kind of a mess and it's very hard to continue it. The problems I see are: * Porting DMD source to Java was done manually and it's a very boring and long task, and we need to find a way to automate it if we'd like to support really good integration with the language (I mean, real semantic value, and because D is not dynamic I think this is worth it).

I have two suggestions for this problem: 1. Could DMD be compiled to a dynamic library and then be used like a plugin, using JNI to interact between the compiler and the plugin.

Nope, the compiler generates a big structure, an AST, which is composed of nodes from a complex hierarchy of classes. Transferring such sctructure across JNI would be incredibly hard do implement, not to mention probably inefficient. JNI (as with most any C interfacing) is good mostly just for calling C methods with simple parameters. The other option would be for the compiler to expose just a thin API (without big data structures), and have the IDE query the semantic functionality directly to the compiler. But then the frontend itself would have to be extended a lot, which would me a lot of complicated coding in C... argh, no way...

Is working with ddmd an option?

Nah. Since DDMD is a direct port of DMD, the situation would not be much different: you would have a lot of C-like code in D, as well as a Walter-like design :P, and you wouldn't be able to change either of those things, because then you would make it significantly hard (if not impractical) to update the DDMD port with new DMD changes. That's exactly the same problem there is with the Java port of DMD. I'm now pretty sure the only good long-term strategy is to have a compiler/engine designed from scratch to work with IDE functionality. Although I didnt realize until now, thanks to your comment, that it might be feasible for this engine to be separate, (either as a process, or as library), from the Java IDE, and thus the engine could be in a different language than Java. Hum, this is quite interesting, one very big reason being that then it could be reused by different D IDEs, or other sorts of projects...

That, and being written in D, would make it more attractive. There is another option with a headstart: dil, if gpl is not a problem.

GPL is generally a problem with Eclipse since the Eclipse license (EPL) is incompatible with GPL. The nature of GPL is that you cannot even dynamically link to a GPL library without effecting the license of your own code and third party code you use. Basically the only way around that is inter-process communication. The CDT pluing for Eclipse uses this method to communicate with GCC. -- /Jacob Carlborg
Jun 15 2010
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2010-05-25 04.15, Ary Borenszweig wrote:
 On 05/24/2010 08:38 AM, Bruno Medeiros wrote:
 Hi.
 ...


 * Porting DMD source to Java was done manually and it's a very boring
 and long task, and we need to find a way to automate it if we'd like to
 support really good integration with the language (I mean, real semantic
 value, and because D is not dynamic I think this is worth it).
 * The DMD source was modified a little bit for performance reasons and
 for integration with Descent so now it contains bugs that are very hard
 to fix.
 * The code is just too big because it has a lot of code from JDT,
 modified a little bit, and that makes it hard for other developers to join.

The best would be, for Descent's point of view, to do as Eclipse did: create a new incremental compiler completely written in Java. But this is easier said than done. -- /Jacob Carlborg
May 25 2010
prev sibling next sibling parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 25/05/2010 03:15, Ary Borenszweig wrote:
 On 05/24/2010 08:38 AM, Bruno Medeiros wrote:

 * Porting DMD source to Java was done manually and it's a very boring
 and long task, and we need to find a way to automate it if we'd like to
 support really good integration with the language (I mean, real semantic
 value, and because D is not dynamic I think this is worth it).
 * The DMD source was modified a little bit for performance reasons and
 for integration with Descent so now it contains bugs that are very hard
 to fix.

That's actually exactly the concerns I had from the start with doing that approach. Even if we had an easy way to automate the port, I think we would still have a code that has issues of performance and scalability. And yet if we would try to modify it to address those issues, we would most likely introduce hard to correct bugs, and/or make the automated porting process (if we had one) much more complicated, if not impossible (aka, reverting to doing a lot of manual work). That's why I think the approach of porting the full DMD frontend is not good, even it was more automated. So for me its either: * Port and use the DMD parser only, and implement semantic analysys features by scratch. This is Mmrnmhrm's current approach, and would mean the semantic capabilities would be much less than Descent: likely just resolving references (for code completion), but no semantic errors or CTFE debug. * Try and build the parser and semantic engine with the help of some other tool, like IMP for example. I'll have to take a look at IMP again to think about this some more.
 * The code is just too big because it has a lot of code from JDT,
 modified a little bit, and that makes it hard for other developers to join.

 .. This means I'm currently
 the only Eclipse developer, so I could just do things in the way I'm
 thinking of, but Ary, I would still like to get your input, because I
 hope one day you might be interested in coming back to Descent
 development. :)

I can program now and then if I have time, but what I'd really like is to plan how to start things almost from scratch and think of a plan of doing it well. Maybe Descent could be done with IMP, I don't think using JDT's source code will be good for the project. The only problem I see with IMP (or DLTK) is that they don't support some of the features Descent already provides... but as IMP gets better Descent will automatically get those updates.

Yes, this is the other issue is how to do all the IDE infrastructure stuff. I also think that porting JDT is less than ideal. Porting JDT has similar problems to the DMD approach (although to a much lesser degree). The code is not easy to understand, and we risk introducing bugs. And it is also hard to maintain and keep updated with new JDT versions. If there was no project such as IMP or DLTK, I think it would still be worth to do the JDT porting. But since those projects exist and are currently actively developed, I think they are a much better choice. In the cases where they don't provide the features that Descent already supports, I think that we might be able to do a little bit of mix-and-match, and use a some of JDT's ported code. But the feasibility of that will depend a lot on the particular feature, and whether IMP or DLTK is used. As for which to use, I'm still inclined torwards DLTK. DLTK has more IDE infrastructure support and functionality than IMP, and might make it easier to mix-and-match with JDT porting, since a lot of DLTK itself is also based on JDT ported code. We might even submit contributions to DLTK itself. But it has no support for parsing or language semantic analysis, which IMP has. So that's an advantage for IMP. But I'll have to take another look at IMP (an also the new DLTK version, since I'm not yet familiar with all the changes that have occurred in the last year or so).
 Ary, I would like to know if by any chance you are interested in doing
 that separation in the near future. If not, I can do it myself, but it
 might take a bit longer.

Yes, I would like to do it, but first I think we need to think if we want to a) keep using the Java port and little by little upgrade it to the latest dmd versions, b) start another port but build a tool to more or less automate that process (Frank Benoit started something in that regard but I don't know what happened then) or c) make a less semantic-aware IDE like Mmrnmhrm (the good thing is that it might be simpler and faster, but the bad thing is that we might not get all the features we'd want). Thanks for bringing this topic! A lot of effort has been put in Descent and it would maybe be sad if the project is abandoned... (but I really feel I'm stuck now and I don't know how to advance... or at least I know I would have to keep porting C++ code to Java, but... http://www.explosm.net/comics/1804/ ) Cheers!

Cool to see you a bit more re-motivated. :) So let me say what I want to do next: Mmrnmrhm at the moment has no support for D2 whatsoever (well its not completely unusable, one can open D2 files but it will treat them as D1 files). I want to take the current Descent parser, and update Mmrnmrhm's semantic engine to this new parser version, such that Mmrnmrhm's current semantic functionality (go to references, find references) works in D2 as well. While doing this, it should give a good idea of how complex it is to keep up the basic IDE functionality on par with language changes, with the approach of implementing the code by scratch. I'm confident this can be done without much effort, and thus it would validate itself as the best approach. -- Bruno Medeiros - Software Engineer
May 25 2010
next sibling parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 05/25/2010 03:31 PM, Bruno Medeiros wrote:
 On 25/05/2010 03:15, Ary Borenszweig wrote:
 On 05/24/2010 08:38 AM, Bruno Medeiros wrote:

 * Porting DMD source to Java was done manually and it's a very boring
 and long task, and we need to find a way to automate it if we'd like to
 support really good integration with the language (I mean, real semantic
 value, and because D is not dynamic I think this is worth it).
 * The DMD source was modified a little bit for performance reasons and
 for integration with Descent so now it contains bugs that are very hard
 to fix.

That's actually exactly the concerns I had from the start with doing that approach. Even if we had an easy way to automate the port, I think we would still have a code that has issues of performance and scalability. And yet if we would try to modify it to address those issues, we would most likely introduce hard to correct bugs, and/or make the automated porting process (if we had one) much more complicated, if not impossible (aka, reverting to doing a lot of manual work). That's why I think the approach of porting the full DMD frontend is not good, even it was more automated. So for me its either: * Port and use the DMD parser only, and implement semantic analysys features by scratch. This is Mmrnmhrm's current approach, and would mean the semantic capabilities would be much less than Descent: likely just resolving references (for code completion), but no semantic errors or CTFE debug. * Try and build the parser and semantic engine with the help of some other tool, like IMP for example.

If anyone cares, I have a functioning ANTLR grammar for d1 for which I someday intend to write semantic for. I'm also mostly done with an ANTLR grammar for d2. They're a bit on the slow side, but I think they can successfully parse things that dmd currently can't.
May 26 2010
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 26/05/2010 16:45, Ellery Newcomer wrote:
 On 05/25/2010 03:31 PM, Bruno Medeiros wrote:
 On 25/05/2010 03:15, Ary Borenszweig wrote:
 On 05/24/2010 08:38 AM, Bruno Medeiros wrote:

 * Porting DMD source to Java was done manually and it's a very boring
 and long task, and we need to find a way to automate it if we'd like to
 support really good integration with the language (I mean, real semantic
 value, and because D is not dynamic I think this is worth it).
 * The DMD source was modified a little bit for performance reasons and
 for integration with Descent so now it contains bugs that are very hard
 to fix.

That's actually exactly the concerns I had from the start with doing that approach. Even if we had an easy way to automate the port, I think we would still have a code that has issues of performance and scalability. And yet if we would try to modify it to address those issues, we would most likely introduce hard to correct bugs, and/or make the automated porting process (if we had one) much more complicated, if not impossible (aka, reverting to doing a lot of manual work). That's why I think the approach of porting the full DMD frontend is not good, even it was more automated. So for me its either: * Port and use the DMD parser only, and implement semantic analysys features by scratch. This is Mmrnmhrm's current approach, and would mean the semantic capabilities would be much less than Descent: likely just resolving references (for code completion), but no semantic errors or CTFE debug. * Try and build the parser and semantic engine with the help of some other tool, like IMP for example.

If anyone cares, I have a functioning ANTLR grammar for d1 for which I someday intend to write semantic for. I'm also mostly done with an ANTLR grammar for d2.

Yes, that might be quite interesting. With that, if going with the Mmrnmhrm approach, we could replace the DMD parser with the ANTLR one, which could be faster overall. Although the DMD parsing itself is likely quite fast, the thing is Mmrnmhrm converts the DMD AST to its own AST structure, which has quite a different class hierarchy. With the ANTRL parser this new AST could be created straight away, which might be faster overall. But this would have to be tested. In any case, its not something for the near future, but its good to know.
 They're a bit on the slow side, but I think they can successfully parse
 things that dmd currently can't.

What do mean slow side? And what things can it parse that DMD can't? (I wasn't aware there were things DMD could not parse) -- Bruno Medeiros - Software Engineer
May 28 2010
parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 05/28/2010 05:49 AM, Bruno Medeiros wrote:
 What do mean slow side? And what things can it parse that DMD can't? (I
 wasn't aware there were things DMD could not parse)

There's a few corner cases. eg: int delegate()[] i = [{while(true){}}]; I think they're all in bugzilla. If you want a gauge on speed, it usually takes 10 to 15 seconds to parse through all of tango's code. (As an LL(k) parser, ANTLR requires lookahead in places that dmd doesn't.)
May 28 2010
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 28/05/2010 15:44, Ellery Newcomer wrote:
 On 05/28/2010 05:49 AM, Bruno Medeiros wrote:
 What do mean slow side? And what things can it parse that DMD can't? (I
 wasn't aware there were things DMD could not parse)

There's a few corner cases. eg: int delegate()[] i = [{while(true){}}]; I think they're all in bugzilla. If you want a gauge on speed, it usually takes 10 to 15 seconds to parse through all of tango's code. (As an LL(k) parser, ANTLR requires lookahead in places that dmd doesn't.)

BTW, what project is that parser for? Is it internal (like commercial) or is open source? Is it listed here: http://prowiki.org/wiki4d/wiki.cgi?GrammarParsers ? -- Bruno Medeiros - Software Engineer
Jun 03 2010
parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 06/03/2010 08:36 AM, Bruno Medeiros wrote:
 On 28/05/2010 15:44, Ellery Newcomer wrote:
 On 05/28/2010 05:49 AM, Bruno Medeiros wrote:
 What do mean slow side? And what things can it parse that DMD can't? (I
 wasn't aware there were things DMD could not parse)

There's a few corner cases. eg: int delegate()[] i = [{while(true){}}]; I think they're all in bugzilla. If you want a gauge on speed, it usually takes 10 to 15 seconds to parse through all of tango's code. (As an LL(k) parser, ANTLR requires lookahead in places that dmd doesn't.)

BTW, what project is that parser for? Is it internal (like commercial) or is open source? Is it listed here: http://prowiki.org/wiki4d/wiki.cgi?GrammarParsers ?

Hm. Never knew about that page :) There might be an old version of it in antlrd; I'll check; but it's a project I started a year or two ago with the intent to eventually build an IDE with.
Jun 03 2010
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 03/06/2010 14:59, Ellery Newcomer wrote:
 On 06/03/2010 08:36 AM, Bruno Medeiros wrote:
 On 28/05/2010 15:44, Ellery Newcomer wrote:
 On 05/28/2010 05:49 AM, Bruno Medeiros wrote:
 What do mean slow side? And what things can it parse that DMD can't? (I
 wasn't aware there were things DMD could not parse)

There's a few corner cases. eg: int delegate()[] i = [{while(true){}}]; I think they're all in bugzilla. If you want a gauge on speed, it usually takes 10 to 15 seconds to parse through all of tango's code. (As an LL(k) parser, ANTLR requires lookahead in places that dmd doesn't.)

BTW, what project is that parser for? Is it internal (like commercial) or is open source? Is it listed here: http://prowiki.org/wiki4d/wiki.cgi?GrammarParsers ?

Hm. Never knew about that page :) There might be an old version of it in antlrd; I'll check; but it's a project I started a year or two ago with the intent to eventually build an IDE with.

Could you elaborate a bit more? I think I read in another post that your parser targets the Java language, I didnt expect that, so the IDE you're planning is in Java as well?? Would that be for a Netbeans based IDE, or is it something commercial you are thinking? Basicly I'm wondering why you wouldnt join the Descent project, if is in Java as well :) -- Bruno Medeiros - Software Engineer
Jun 15 2010
parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 06/15/2010 08:16 AM, Bruno Medeiros wrote:
 On 03/06/2010 14:59, Ellery Newcomer wrote:
 On 06/03/2010 08:36 AM, Bruno Medeiros wrote:
 On 28/05/2010 15:44, Ellery Newcomer wrote:
 On 05/28/2010 05:49 AM, Bruno Medeiros wrote:
 What do mean slow side? And what things can it parse that DMD
 can't? (I
 wasn't aware there were things DMD could not parse)

There's a few corner cases. eg: int delegate()[] i = [{while(true){}}]; I think they're all in bugzilla. If you want a gauge on speed, it usually takes 10 to 15 seconds to parse through all of tango's code. (As an LL(k) parser, ANTLR requires lookahead in places that dmd doesn't.)

BTW, what project is that parser for? Is it internal (like commercial) or is open source? Is it listed here: http://prowiki.org/wiki4d/wiki.cgi?GrammarParsers ?

Hm. Never knew about that page :) There might be an old version of it in antlrd; I'll check; but it's a project I started a year or two ago with the intent to eventually build an IDE with.

Could you elaborate a bit more? I think I read in another post that your parser targets the Java language, I didnt expect that, so the IDE you're planning is in Java as well?? Would that be for a Netbeans based IDE, or is it something commercial you are thinking? Basicly I'm wondering why you wouldnt join the Descent project, if is in Java as well :)

Yeah, netbeans. Currently, my semantic analysis is all vaporware. The parser is slow, and incomplete (unicode: surrogate pairs - antlr sucks), and not yet set up for being amenable for things like syntax highlighting. And I have a habit of working feverishly on it and then leaving it to rot for months. I never joined Descent because I don't care much for eclipse and Ary's choice to port dmd didn't agree with me. But if you're going to be redesigning semantic from scratch, and you're going to leave the door open to port Descent to netbeans, sure, I'd love to contribute.
Jun 15 2010
parent Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 15/06/2010 14:43, Ellery Newcomer wrote:
 On 06/15/2010 08:16 AM, Bruno Medeiros wrote:
 On 03/06/2010 14:59, Ellery Newcomer wrote:
 On 06/03/2010 08:36 AM, Bruno Medeiros wrote:
 On 28/05/2010 15:44, Ellery Newcomer wrote:
 On 05/28/2010 05:49 AM, Bruno Medeiros wrote:
 What do mean slow side? And what things can it parse that DMD
 can't? (I
 wasn't aware there were things DMD could not parse)

There's a few corner cases. eg: int delegate()[] i = [{while(true){}}]; I think they're all in bugzilla. If you want a gauge on speed, it usually takes 10 to 15 seconds to parse through all of tango's code. (As an LL(k) parser, ANTLR requires lookahead in places that dmd doesn't.)

BTW, what project is that parser for? Is it internal (like commercial) or is open source? Is it listed here: http://prowiki.org/wiki4d/wiki.cgi?GrammarParsers ?

Hm. Never knew about that page :) There might be an old version of it in antlrd; I'll check; but it's a project I started a year or two ago with the intent to eventually build an IDE with.

Could you elaborate a bit more? I think I read in another post that your parser targets the Java language, I didnt expect that, so the IDE you're planning is in Java as well?? Would that be for a Netbeans based IDE, or is it something commercial you are thinking? Basicly I'm wondering why you wouldnt join the Descent project, if is in Java as well :)

Yeah, netbeans. Currently, my semantic analysis is all vaporware. The parser is slow, and incomplete (unicode: surrogate pairs - antlr sucks), and not yet set up for being amenable for things like syntax highlighting. And I have a habit of working feverishly on it and then leaving it to rot for months. I never joined Descent because I don't care much for eclipse and Ary's choice to port dmd didn't agree with me. But if you're going to be redesigning semantic from scratch, and you're going to leave the door open to port Descent to netbeans, sure, I'd love to contribute.

Porting the whole of Descent to Netbeans doesn't make any sense, one would need to port Eclipse to Netbeans... :P But as for the semantic engine, it does make sense: my idea is not just to separate it from the rest of descent.core (like I mentioned previously), but also to make it independent of Eclipse. Thus it becomes trivially resuable in any other Java project like a Netbeans IDE. Whether I'll try to rewrite the parser, or keep using DMD's parser, that's something that would have to be thought about later. -- Bruno Medeiros - Software Engineer
Jun 15 2010
prev sibling parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 25/05/2010 21:31, Bruno Medeiros wrote:
 On 25/05/2010 03:15, Ary Borenszweig wrote:
 On 05/24/2010 08:38 AM, Bruno Medeiros wrote:

 * Porting DMD source to Java was done manually and it's a very boring
 and long task, and we need to find a way to automate it if we'd like to
 support really good integration with the language (I mean, real semantic
 value, and because D is not dynamic I think this is worth it).
 * The DMD source was modified a little bit for performance reasons and
 for integration with Descent so now it contains bugs that are very hard
 to fix.

That's actually exactly the concerns I had from the start with doing that approach. Even if we had an easy way to automate the port, I think we would still have a code that has issues of performance and scalability. And yet if we would try to modify it to address those issues, we would most likely introduce hard to correct bugs, and/or make the automated porting process (if we had one) much more complicated, if not impossible (aka, reverting to doing a lot of manual work). That's why I think the approach of porting the full DMD frontend is not good, even it was more automated. So for me its either: * Port and use the DMD parser only, and implement semantic analysys features by scratch. This is Mmrnmhrm's current approach, and would mean the semantic capabilities would be much less than Descent: likely just resolving references (for code completion), but no semantic errors or CTFE debug. * Try and build the parser and semantic engine with the help of some other tool, like IMP for example. I'll have to take a look at IMP again to think about this some more.
 * The code is just too big because it has a lot of code from JDT,
 modified a little bit, and that makes it hard for other developers to
 join.

 .. This means I'm currently
 the only Eclipse developer, so I could just do things in the way I'm
 thinking of, but Ary, I would still like to get your input, because I
 hope one day you might be interested in coming back to Descent
 development. :)

I can program now and then if I have time, but what I'd really like is to plan how to start things almost from scratch and think of a plan of doing it well. Maybe Descent could be done with IMP, I don't think using JDT's source code will be good for the project. The only problem I see with IMP (or DLTK) is that they don't support some of the features Descent already provides... but as IMP gets better Descent will automatically get those updates.

Yes, this is the other issue is how to do all the IDE infrastructure stuff. I also think that porting JDT is less than ideal. Porting JDT has similar problems to the DMD approach (although to a much lesser degree). The code is not easy to understand, and we risk introducing bugs. And it is also hard to maintain and keep updated with new JDT versions. If there was no project such as IMP or DLTK, I think it would still be worth to do the JDT porting. But since those projects exist and are currently actively developed, I think they are a much better choice. In the cases where they don't provide the features that Descent already supports, I think that we might be able to do a little bit of mix-and-match, and use a some of JDT's ported code. But the feasibility of that will depend a lot on the particular feature, and whether IMP or DLTK is used. As for which to use, I'm still inclined torwards DLTK. DLTK has more IDE infrastructure support and functionality than IMP, and might make it easier to mix-and-match with JDT porting, since a lot of DLTK itself is also based on JDT ported code. We might even submit contributions to DLTK itself. But it has no support for parsing or language semantic analysis, which IMP has. So that's an advantage for IMP. But I'll have to take another look at IMP (an also the new DLTK version, since I'm not yet familiar with all the changes that have occurred in the last year or so).
 Ary, I would like to know if by any chance you are interested in doing
 that separation in the near future. If not, I can do it myself, but it
 might take a bit longer.

Yes, I would like to do it, but first I think we need to think if we want to a) keep using the Java port and little by little upgrade it to the latest dmd versions, b) start another port but build a tool to more or less automate that process (Frank Benoit started something in that regard but I don't know what happened then) or c) make a less semantic-aware IDE like Mmrnmhrm (the good thing is that it might be simpler and faster, but the bad thing is that we might not get all the features we'd want). Thanks for bringing this topic! A lot of effort has been put in Descent and it would maybe be sad if the project is abandoned... (but I really feel I'm stuck now and I don't know how to advance... or at least I know I would have to keep porting C++ code to Java, but... http://www.explosm.net/comics/1804/ ) Cheers!

Cool to see you a bit more re-motivated. :) So let me say what I want to do next: Mmrnmrhm at the moment has no support for D2 whatsoever (well its not completely unusable, one can open D2 files but it will treat them as D1 files). I want to take the current Descent parser, and update Mmrnmrhm's semantic engine to this new parser version, such that Mmrnmrhm's current semantic functionality (go to references, find references) works in D2 as well. While doing this, it should give a good idea of how complex it is to keep up the basic IDE functionality on par with language changes, with the approach of implementing the code by scratch. I'm confident this can be done without much effort, and thus it would validate itself as the best approach.

Ok, status update. I've done the above, updated Mmrnmrhm using Descent's parser. It now works ok (same functionality as Mmrnmrhm was with D1.0) for D2, well, at least up to D #2.030. (you get syntax errors with annotations, for example) It was fairly quick task, it only took 2 man-days. I stick by my conclusion that it is fairly easy to keep a parser up-to-date with language changes, together with a basic custom IDE semantic engine to provide reference resolution and search (which is the core of the features of proper code completion, find reference, hover text, etc.). That's not surprising since these aspects of a language rarely change. Other aspects of the language not directly related to reference resolution, but that affect it, change more often, but on the other hand are easier to adapt. (and also, if they are not adapted, the IDE is not too crippled) This is in contrast to, for example, maintaining a custom semantic engine for semantic error reporting. This would make it much harder to keep up to date with language changes, and much more crippling to the IDE if not kept up to date (if outdated, one would likely get a lot of false errors in the editor). Anyways, I again took a better look at IMP, and also XText (a project very similar in purpose to IMP), but I'm not convinced they offer a significant advantage over DLTK. Yes, DLTK is only about IDE "infrastructure", its not a complete solution and in particular has almost nothing to support language semantic engine functionality. IMP and XText on the other hand do aim to support this kind of functionality, but in practice I think that this support is somewhat superficial, and that for non-simple languages (aka, not DSLs), you still have to write the bulk of the code for the semantic functionality. In more detail, looking at this quote from IMP intro: "" The goal of the project is to ease the development of commercial-quality IDE support for new programming languages, including the following features: [...] * refactoring support (not only "Move" and "Rename", but type- and code-related refactorings requiring non-trivial analysis, e.g. "Extract Method" and "Infer Type Arguments") "" However this particular support has not yet been implement in IMP, nor has it even show how it could be implemented, and its usage would be. So this particular promise is vaporware at the moment. The only main semantic functionality that IMP provides is reference resolving. But even for that, check this quote from the IMP User Guide, section "Creating a Reference-Resolution Service": "" The customization of this service implementation for a particular language may be simple or complex, depending on language semantics and on the availability of information that can facilitate the resolution process. "" So basically this is exactly the same stuff I did for Mmrnmhrm already. At the moment, not much is gained from using IMP it seems, versus DLTK. I also looked at XText, a more recent project, but which has gained a lot of traction in Eclipse recently. It has even matured as an Eclipse project (like DLTK, but unlike IMP), and has even been featured in the recent Helios release. I've tried out XText, and actually tried coding a very simple language on it. It seems XText actually delivers something on that promise of providing support for core semantic functionality: they have their own grammar language which allows you to annotate and specify certain kinds of semantic information, namely regarding error validation/reporting and reference resolving. With XText it is actually possible to build an IDE/editor with reference resolving (that means Code Completion, Find Reference, etc.) without writing any code for something like a ReferenceResolver. Just annotate the language grammar to specify what particular nodes/productions are references (and to what kind of entity it references). I find that XText is quite a step ahead of IMP, semantics-wise. However, I still don't think it would be worthwhile to adopt XText. It works great for simple and moderately complex DSLs, but for a full blown general purpose language like D (which is quite complex, even amongst GPLs), I think that support would erode, and you would have to write the bulk of the semantic code anyways, and have little to gain. One thing of interest though, is their use of EMF: the ASTs that XText generates are based on EMF, and they claim that this provides them with a lot of tools and functionality for manipulating the AST using generic EMF tooling. I haven't looked into this with depth, but it seems potentially substantial and worthwhile to keep in mind, should we ever get to point of doing AST manipulations (for refactoring, etc.). So, yeah, as things are, I'm maintaining my approach with Mmrnmrhm: writing and improving the (very basic) semantic engine from scratch, and using DLTK. As for the parser, it will continue to use Descent's parser, although the parser is already a bit outdated for D2. :/ It would be great if someone were to volunteer to update it... *wink* *wink* :p But better yet would be to start coding our own custom parser (using a parser generator like ANTLR for example), that could really be tailored for IDE needs. In the medium/long term, that's probably what needs to be done. -- Bruno Medeiros - Software Engineer
Jul 12 2010
parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 07/12/2010 02:45 PM, Bruno Medeiros wrote:
 So, yeah, as things are, I'm maintaining my approach with Mmrnmrhm:
 writing and improving the (very basic) semantic engine from scratch, and
 using DLTK.
 As for the parser, it will continue to use Descent's parser, although
 the parser is already a bit outdated for D2. :/ It would be great if
 someone were to volunteer to update it... *wink* *wink* :p

sigh, point me at the source
 But better yet would be to start coding our own custom parser (using a
 parser generator like ANTLR for example), that could really be tailored
 for IDE needs. In the medium/long term, that's probably what needs to be
 done.

What in particular does an IDE need tailored to it?
Jul 12 2010
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 12/07/2010 21:31, Ellery Newcomer wrote:
 On 07/12/2010 02:45 PM, Bruno Medeiros wrote:
 So, yeah, as things are, I'm maintaining my approach with Mmrnmrhm:
 writing and improving the (very basic) semantic engine from scratch, and
 using DLTK.
 As for the parser, it will continue to use Descent's parser, although
 the parser is already a bit outdated for D2. :/ It would be great if
 someone were to volunteer to update it... *wink* *wink* :p

sigh, point me at the source

Great, but are you aware what this task entails? In other words, are you familiar with descent.compiler? Because this is a Java port of the DMD frontend, and I'm not sure how easy it is for someone else other than Ary to learn how to do this, there might be quite a bit of private knowledge. (we can ask him about this, of course)
 But better yet would be to start coding our own custom parser (using a
 parser generator like ANTLR for example), that could really be tailored
 for IDE needs. In the medium/long term, that's probably what needs to be
 done.

What in particular does an IDE need tailored to it?

Well, in terms of *pure functionality*, there is not much else an IDE needs, other than making sure it can get all source information (like node source text ranges, comments, etc.) The big thing that can be tailored is performance: incremental parsing, being able to do different kinds of parsing with different levels of detail ("sub-parsing"?). Some of these performance features become very important for IDE scalability. Good error-recovery is also important, although that is also usually useful for a compiler (even if not so much as an IDE). -- Bruno Medeiros - Software Engineer
Jul 14 2010
next sibling parent Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 14/07/2010 11:46, Bruno Medeiros wrote:
 On 12/07/2010 21:31, Ellery Newcomer wrote:
 On 07/12/2010 02:45 PM, Bruno Medeiros wrote:
 So, yeah, as things are, I'm maintaining my approach with Mmrnmrhm:
 writing and improving the (very basic) semantic engine from scratch, and
 using DLTK.
 As for the parser, it will continue to use Descent's parser, although
 the parser is already a bit outdated for D2. :/ It would be great if
 someone were to volunteer to update it... *wink* *wink* :p

sigh, point me at the source

Great, but are you aware what this task entails? In other words, are you familiar with descent.compiler? Because this is a Java port of the DMD frontend, and I'm not sure how easy it is for someone else other than Ary to learn how to do this, there might be quite a bit of private knowledge. (we can ask him about this, of course)
 But better yet would be to start coding our own custom parser (using a
 parser generator like ANTLR for example), that could really be tailored
 for IDE needs. In the medium/long term, that's probably what needs to be
 done.



Ah, forgot to say, source is: http://www.dsource.org/projects/descent/browser/trunk/descent.compiler -- Bruno Medeiros - Software Engineer
Jul 14 2010
prev sibling parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 07/14/2010 05:46 AM, Bruno Medeiros wrote:
 On 12/07/2010 21:31, Ellery Newcomer wrote:
 On 07/12/2010 02:45 PM, Bruno Medeiros wrote:
 So, yeah, as things are, I'm maintaining my approach with Mmrnmrhm:
 writing and improving the (very basic) semantic engine from scratch, and
 using DLTK.
 As for the parser, it will continue to use Descent's parser, although
 the parser is already a bit outdated for D2. :/ It would be great if
 someone were to volunteer to update it... *wink* *wink* :p

sigh, point me at the source

Great, but are you aware what this task entails? In other words, are you familiar with descent.compiler? Because this is a Java port of the DMD frontend, and I'm not sure how easy it is for someone else other than Ary to learn how to do this, there might be quite a bit of private knowledge. (we can ask him about this, of course)

Nope, but it doesn't look too much different from dmd's source. I am, however, intimidated by large codebases wrt build (particularly java based ones).
 But better yet would be to start coding our own custom parser (using a
 parser generator like ANTLR for example), that could really be tailored
 for IDE needs. In the medium/long term, that's probably what needs to be
 done.

What in particular does an IDE need tailored to it?

Well, in terms of *pure functionality*, there is not much else an IDE needs, other than making sure it can get all source information (like node source text ranges, comments, etc.) The big thing that can be tailored is performance: incremental parsing, being able to do different kinds of parsing with different levels of detail ("sub-parsing"?). Some of these performance features become very important for IDE scalability. Good error-recovery is also important, although that is also usually useful for a compiler (even if not so much as an IDE).

hmm, for incremental parsing, ANLTR isn't going to be your friend
Jul 14 2010
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 14/07/2010 14:50, Ellery Newcomer wrote:
 I am, however, intimidated by large codebases wrt build (particularly
 java based ones).

What do you mean with regards to build? -- Bruno Medeiros - Software Engineer
Jul 19 2010
parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 07/19/2010 11:13 AM, Bruno Medeiros wrote:
 On 14/07/2010 14:50, Ellery Newcomer wrote:
 I am, however, intimidated by large codebases wrt build (particularly
 java based ones).

What do you mean with regards to build?

I mean I might need some help setting things up. I haven't tried yet, and I actually have time today, so I'll see if I can't figure things out.
Jul 19 2010
parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 19/07/2010 21:42, Ellery Newcomer wrote:
 On 07/19/2010 11:13 AM, Bruno Medeiros wrote:
 On 14/07/2010 14:50, Ellery Newcomer wrote:
 I am, however, intimidated by large codebases wrt build (particularly
 java based ones).

What do you mean with regards to build?

I mean I might need some help setting things up. I haven't tried yet, and I actually have time today, so I'll see if I can't figure things out.

Shouldn't be too complicated. You only need to checkout the descent.compiler project, but you will need to run it with Eclipse+PDE, because descent.compiler does have some dependencies on Eclipse code (I want to remove that eventually, shouldn't be too hard). The dtool project might be of interested as well, I am moving there some tests there that also exercise the parser in descent.compiler. -- Bruno Medeiros - Software Engineer
Jul 21 2010
parent reply Ellery Newcomer <ellery-newcomer utulsa.edu> writes:
On 07/21/2010 01:15 PM, Bruno Medeiros wrote:
 On 19/07/2010 21:42, Ellery Newcomer wrote:
 On 07/19/2010 11:13 AM, Bruno Medeiros wrote:
 On 14/07/2010 14:50, Ellery Newcomer wrote:
 I am, however, intimidated by large codebases wrt build (particularly
 java based ones).

What do you mean with regards to build?

I mean I might need some help setting things up. I haven't tried yet, and I actually have time today, so I'll see if I can't figure things out.

Shouldn't be too complicated. You only need to checkout the descent.compiler project, but you will need to run it with Eclipse+PDE, because descent.compiler does have some dependencies on Eclipse code (I want to remove that eventually, shouldn't be too hard). The dtool project might be of interested as well, I am moving there some tests there that also exercise the parser in descent.compiler.

hey Bruno, is there a way to contact you if we want to chat but not on the DMD newsgroup?
Aug 23 2010
parent Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
On 23/08/2010 20:13, Ellery Newcomer wrote:
 hey Bruno, is there a way to contact you if we want to chat but not on
 the DMD newsgroup?

Hi Ellery. Sorry for the late reply, I was (and still am) on holidays. You can contact me on Google Chat on: bruno DOT do DOT medeiros ATgmailDOTcom or alternatively on MSN: dai DOT phoenix ATgmailDOTcom I'm usually logged in (and available) on the first IM when working, and on the second one the rest of the time. -- Bruno Medeiros - Software Engineer
Sep 02 2010
prev sibling parent Adrian <nmanole gmail.com> writes:
Maybe it would be useful to take a look at Object Teams
(http://www.eclipse.org/objectteams) for some of the necessary rework. This
article,
http://contraptionsforprogramming.blogspot.com/2010/02/approaches-to-extending-jdt.html,
discusses a (similar?) problem of needing to extend the JDT for Groovy support.

Cheers,
Adrian
Jun 03 2010