www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - dflect alpha

reply "Ben Hinkle" <ben.hinkle gmail.com> writes:
I've started poking around with a reflection API for D called "dflect" and 
here's what I have after a day (the zip of the files is 
http://home.comcast.net/~benhinkle/dflect.zip). Comments welcome.
The basic idea is that to generate metadata about a module you run 
dflect.exe (based not surprisingly on dmdfe) which takes a module foo and 
generates another D source file fooinfo with auto-generated classes 
describing the original module. You then compile and link in these metadata 
modules into the final binary. Right now it only supports fields but I'll 
start poking around with methods next. I haven't even started thinking about 
templates and more complex modules. For example given the module

module test;
class Foo {
  int x;
  char[] y;
}

running dflect gives something like

// Auto-generated reflection data for module test
import dflect;
import test;
class FooInfo : MetaClass
{
    static MetaClass makeInfo(){return new FooInfo;}
    Object construct(){return new Foo;}
    this() {
      fields["x"] = new MetaField("x",typeid(int),8,4);
      fields["y"] = new MetaField("y",typeid(char[]),12,8);
    }
}
static this() {
  dflect.register("test.Foo",&FooInfo.makeInfo);
}

Here's a sample app using the resulting metadata:

import dflect;
import test;
int main() {
  MetaClass cls = getInfo("test.Foo");
  Foo u = new Foo;
  MetaField field = cls.fields["x"];
  field.set(u,100);
  field = cls.fields["y"];
  field.set(u,"hello");
  printf("setter %d %.*s\n",u.x,u.y);
  char[] res;
  field.get(u,&res);
  printf("getter %.*s\n",res);
  return 0;
}

C:\dflect\dflect>dflect test.d
C:\dflect\dflect>dmd dtest.d dflect.d testinfo.d test.d
C:\dflect\dflect>dtest
setter 100 hello
getter hello 
Jul 04 2005
next sibling parent reply "Andrew Fedoniouk" <news terrainformatica.com> writes:
Cool!

One question: as far as I remember order
of fields in class instances may be different from
declaration order.

How then this will reliably work for classes?:

      fields["x"] = new MetaField("x",typeid(int),8,4);
      fields["y"] = new MetaField("y",typeid(char[]),12,8);

I mean that dflect.exe must be always used with the same set of command line switches as used with dmd.exe and version of dflect.exe must be always in sync with dmd.exe. Am I right? BTW: does D class layout depends on command line switches? Andrew. "Ben Hinkle" <ben.hinkle gmail.com> wrote in message news:dab8av$s2q$1 digitaldaemon.com...
 I've started poking around with a reflection API for D called "dflect" and 
 here's what I have after a day (the zip of the files is 
 http://home.comcast.net/~benhinkle/dflect.zip). Comments welcome.
 The basic idea is that to generate metadata about a module you run 
 dflect.exe (based not surprisingly on dmdfe) which takes a module foo and 
 generates another D source file fooinfo with auto-generated classes 
 describing the original module. You then compile and link in these 
 metadata modules into the final binary. Right now it only supports fields 
 but I'll start poking around with methods next. I haven't even started 
 thinking about templates and more complex modules. For example given the 
 module

 module test;
 class Foo {
  int x;
  char[] y;
 }

 running dflect gives something like

 // Auto-generated reflection data for module test
 import dflect;
 import test;
 class FooInfo : MetaClass
 {
    static MetaClass makeInfo(){return new FooInfo;}
    Object construct(){return new Foo;}
    this() {
      fields["x"] = new MetaField("x",typeid(int),8,4);
      fields["y"] = new MetaField("y",typeid(char[]),12,8);
    }
 }
 static this() {
  dflect.register("test.Foo",&FooInfo.makeInfo);
 }

 Here's a sample app using the resulting metadata:

 import dflect;
 import test;
 int main() {
  MetaClass cls = getInfo("test.Foo");
  Foo u = new Foo;
  MetaField field = cls.fields["x"];
  field.set(u,100);
  field = cls.fields["y"];
  field.set(u,"hello");
  printf("setter %d %.*s\n",u.x,u.y);
  char[] res;
  field.get(u,&res);
  printf("getter %.*s\n",res);
  return 0;
 }

 C:\dflect\dflect>dflect test.d
 C:\dflect\dflect>dmd dtest.d dflect.d testinfo.d test.d
 C:\dflect\dflect>dtest
 setter 100 hello
 getter hello
 

Jul 04 2005
next sibling parent reply "Uwe Salomon" <post uwesalomon.de> writes:
 How then this will reliably work for classes?:

      fields["x"] = new MetaField("x",typeid(int),8,4);
      fields["y"] = new MetaField("y",typeid(char[]),12,8);

I mean that dflect.exe must be always used with the same set of command line switches as used with dmd.exe and version of dflect.exe must be always in sync with dmd.exe. Am I right?

I stumbled over that one, too. Isn't it possible to include that like this: fields["x"] = new MetaField("x", typeid(int), Foo.x.offsetof, Foo.x.sizeof); That should always work as expected? Ciao uwe
Jul 04 2005
parent reply "Ben Hinkle" <ben.hinkle gmail.com> writes:
"Uwe Salomon" <post uwesalomon.de> wrote in message 
news:op.stejkmuh6yjbe6 sandmann.maerchenwald.net...
 How then this will reliably work for classes?:

      fields["x"] = new MetaField("x",typeid(int),8,4);
      fields["y"] = new MetaField("y",typeid(char[]),12,8);

I mean that dflect.exe must be always used with the same set of command line switches as used with dmd.exe and version of dflect.exe must be always in sync with dmd.exe. Am I right?

I stumbled over that one, too. Isn't it possible to include that like this: fields["x"] = new MetaField("x", typeid(int), Foo.x.offsetof, Foo.x.sizeof); That should always work as expected? Ciao uwe

You're both right. When I tried Foo.x.offsetof I got a compiler error so I just slapped in the number. I'll post something to D.bugs so that Walter can fix it - assuming what I saw was a bug. I'm also thinking of not importing the original module and in that case the fixed numbers would be the way to go.
Jul 04 2005
parent reply "Ben Hinkle" <bhinkle mathworks.com> writes:
"Ben Hinkle" <ben.hinkle gmail.com> wrote in message 
news:dac0na$1hg5$1 digitaldaemon.com...
 "Uwe Salomon" <post uwesalomon.de> wrote in message 
 news:op.stejkmuh6yjbe6 sandmann.maerchenwald.net...
 How then this will reliably work for classes?:

      fields["x"] = new MetaField("x",typeid(int),8,4);
      fields["y"] = new MetaField("y",typeid(char[]),12,8);

I mean that dflect.exe must be always used with the same set of command line switches as used with dmd.exe and version of dflect.exe must be always in sync with dmd.exe. Am I right?

I stumbled over that one, too. Isn't it possible to include that like this: fields["x"] = new MetaField("x", typeid(int), Foo.x.offsetof, Foo.x.sizeof); That should always work as expected? Ciao uwe

You're both right. When I tried Foo.x.offsetof I got a compiler error so I just slapped in the number. I'll post something to D.bugs so that Walter can fix it - assuming what I saw was a bug. I'm also thinking of not importing the original module and in that case the fixed numbers would be the way to go.

I got method dispatching working and when doing so I needed the slot number in the vtable for each method so there are definitely going to be hard-coded values in the metadata code. I decided to remove the import of the original module so I'm leaving the field offset numbers. The size might get removed and computed from the TypeInfo but for now I'll let it be. I've also written a C header file for all this so now C/C++ code can get/set fields and call methods on D objects using an API that roughly looks like JNI. I'll post an update soon.
Jul 05 2005
parent "Andrew Fedoniouk" <news terrainformatica.com> writes:
 I've also written a C header file for all this so now C/C++ code can 
 get/set
 fields and call methods on D objects using an API that roughly looks like 
 JNI. I'll post an update soon.

Cool. I mean extremely cool. Probably it will be possible to make transparent interop DMDScript <-> D ...
Jul 05 2005
prev sibling parent reply "Ben Hinkle" <ben.hinkle gmail.com> writes:
"Andrew Fedoniouk" <news terrainformatica.com> wrote in message 
news:dabv5q$1gci$1 digitaldaemon.com...
 Cool!

 One question: as far as I remember order
 of fields in class instances may be different from
 declaration order.

 How then this will reliably work for classes?:

      fields["x"] = new MetaField("x",typeid(int),8,4);
      fields["y"] = new MetaField("y",typeid(char[]),12,8);

I mean that dflect.exe must be always used with the same set of command line switches as used with dmd.exe and version of dflect.exe must be always in sync with dmd.exe. Am I right? BTW: does D class layout depends on command line switches?

The only dependency I'm aware of would be version switches. Debugging and DbC switches shouldn't matter. Although to be safe I'd pass the same switches anyway.
Jul 04 2005
parent "Andrew Fedoniouk" <news terrainformatica.com> writes:
"Ben Hinkle" <ben.hinkle gmail.com> wrote in message 
news:dac14g$1ht0$1 digitaldaemon.com...
 "Andrew Fedoniouk" <news terrainformatica.com> wrote in message 
 news:dabv5q$1gci$1 digitaldaemon.com...
 Cool!

 One question: as far as I remember order
 of fields in class instances may be different from
 declaration order.

 How then this will reliably work for classes?:

      fields["x"] = new MetaField("x",typeid(int),8,4);
      fields["y"] = new MetaField("y",typeid(char[]),12,8);

I mean that dflect.exe must be always used with the same set of command line switches as used with dmd.exe and version of dflect.exe must be always in sync with dmd.exe. Am I right? BTW: does D class layout depends on command line switches?

The only dependency I'm aware of would be version switches. Debugging and DbC switches shouldn't matter. Although to be safe I'd pass the same switches anyway.

I see, thanks, Ben.
Jul 04 2005
prev sibling next sibling parent reply Markus Dangl <danglm in.tum.de> writes:
Ben Hinkle schrieb:
 
 C:\dflect\dflect>dflect test.d
 C:\dflect\dflect>dmd dtest.d dflect.d testinfo.d test.d
 C:\dflect\dflect>dtest
 setter 100 hello
 getter hello 
 
 

Sorry, i can't find dflect.d in the zip file. Where can i get it? Markus
Jul 07 2005
parent reply "Ben Hinkle" <ben.hinkle gmail.com> writes:
"Markus Dangl" <danglm in.tum.de> wrote in message 
news:dakp99$273f$1 digitaldaemon.com...
 Ben Hinkle schrieb:
 C:\dflect\dflect>dflect test.d
 C:\dflect\dflect>dmd dtest.d dflect.d testinfo.d test.d
 C:\dflect\dflect>dtest
 setter 100 hello
 getter hello

Sorry, i can't find dflect.d in the zip file. Where can i get it? Markus

I forgot that part. I've updated the zip file and make a page for it http://home.comcast.net/~benhinkle/dflect/ You can call constructors and methods now. Return values are ignored currently. Also most likely lots of little things still don't work.
Jul 07 2005
parent reply pragma <pragma_member pathlink.com> writes:
In article <dakqub$2888$1 digitaldaemon.com>, Ben Hinkle says...
"Markus Dangl" <danglm in.tum.de> wrote in message 
news:dakp99$273f$1 digitaldaemon.com...
 Ben Hinkle schrieb:
 C:\dflect\dflect>dflect test.d
 C:\dflect\dflect>dmd dtest.d dflect.d testinfo.d test.d
 C:\dflect\dflect>dtest
 setter 100 hello
 getter hello

Sorry, i can't find dflect.d in the zip file. Where can i get it? Markus

I forgot that part. I've updated the zip file and make a page for it http://home.comcast.net/~benhinkle/dflect/ You can call constructors and methods now. Return values are ignored currently. Also most likely lots of little things still don't work.

Ben, I had no idea that this was a modified DMD front-end... at first glance it looked like you had merely provided a way to spoof reflection via some other means. Bravo! - EricAnderton at yahoo
Jul 08 2005
parent reply "Ben Hinkle" <bhinkle mathworks.com> writes:
"pragma" <pragma_member pathlink.com> wrote in message 
news:daltti$35e$1 digitaldaemon.com...
 In article <dakqub$2888$1 digitaldaemon.com>, Ben Hinkle says...
"Markus Dangl" <danglm in.tum.de> wrote in message
news:dakp99$273f$1 digitaldaemon.com...
 Ben Hinkle schrieb:
 C:\dflect\dflect>dflect test.d
 C:\dflect\dflect>dmd dtest.d dflect.d testinfo.d test.d
 C:\dflect\dflect>dtest
 setter 100 hello
 getter hello

Sorry, i can't find dflect.d in the zip file. Where can i get it? Markus

I forgot that part. I've updated the zip file and make a page for it http://home.comcast.net/~benhinkle/dflect/ You can call constructors and methods now. Return values are ignored currently. Also most likely lots of little things still don't work.

Ben, I had no idea that this was a modified DMD front-end... at first glance it looked like you had merely provided a way to spoof reflection via some other means. Bravo! - EricAnderton at yahoo

dlint and this reflection generator were the two reasons I wanted dmdfe so I'm just getting around to doing what I wanted to do for a while. I figured the dmd front end does all the work of figuring out the module semantics so why not spit that metadata out? It would also be interesting to spit out the meta data in another format like XML or something instead of spitting out D code. I was going to look into that eventually but if anyone wants to poke around they're certainly welcome to experiment.
Jul 08 2005
parent pragma <pragma_member pathlink.com> writes:
In article <dalvps$4pl$1 digitaldaemon.com>, Ben Hinkle says...
"pragma" <pragma_member pathlink.com> wrote in message 
news:daltti$35e$1 digitaldaemon.com...
 In article <dakqub$2888$1 digitaldaemon.com>, Ben Hinkle says...
"Markus Dangl" <danglm in.tum.de> wrote in message
news:dakp99$273f$1 digitaldaemon.com...
 Ben Hinkle schrieb:
 C:\dflect\dflect>dflect test.d
 C:\dflect\dflect>dmd dtest.d dflect.d testinfo.d test.d
 C:\dflect\dflect>dtest
 setter 100 hello
 getter hello

Sorry, i can't find dflect.d in the zip file. Where can i get it? Markus

I forgot that part. I've updated the zip file and make a page for it http://home.comcast.net/~benhinkle/dflect/ You can call constructors and methods now. Return values are ignored currently. Also most likely lots of little things still don't work.

Ben, I had no idea that this was a modified DMD front-end... at first glance it looked like you had merely provided a way to spoof reflection via some other means. Bravo! - EricAnderton at yahoo

dlint and this reflection generator were the two reasons I wanted dmdfe so I'm just getting around to doing what I wanted to do for a while. I figured the dmd front end does all the work of figuring out the module semantics so why not spit that metadata out? It would also be interesting to spit out the meta data in another format like XML or something instead of spitting out D code. I was going to look into that eventually but if anyone wants to poke around they're certainly welcome to experiment.

Funny you should mention that. I already made an experiment into generating XML with the first DMDFE release you wrote. http://svn.dsource.org/projects/dsp/trunk/dexter/ I'm still learning how to get the most from the front-end. I wanted to use this to write documentation, so it captures comments and associates them with D constructs. Just pass the XML on to a stylesheet engine and presto: instant HTML documentation. I seriously think this is the way to go for producing documentation with D. - EricAnderton at yahoo
Jul 10 2005
prev sibling parent reply "Ben Hinkle" <ben.hinkle gmail.com> writes:
A request for input: what (if any) metadata tags (which aren't already in 
standard D) would people like to see? I'm thinking of C#'s attribute syntax 
and Java's annotations. I suppose the tags will have to be in comments in 
order to get past the D compiler. Should doxygen/javadoc comments be added 
to the metadata? For example I could imagine something very simple where any 
comment preceding a method gets added to the metadata for that method as a 
"description" property. Or it can try to pick apart tags like  doc,  author, 
 version, etc etc.

thoughts? 
Jul 16 2005
parent reply pragma <pragma_member pathlink.com> writes:
In article <dbbcdh$1rv6$1 digitaldaemon.com>, Ben Hinkle says...
A request for input: what (if any) metadata tags (which aren't already in 
standard D) would people like to see? I'm thinking of C#'s attribute syntax 
and Java's annotations. I suppose the tags will have to be in comments in 
order to get past the D compiler. Should doxygen/javadoc comments be added 
to the metadata? For example I could imagine something very simple where any 
comment preceding a method gets added to the metadata for that method as a 
"description" property. Or it can try to pick apart tags like  doc,  author, 
 version, etc etc.

thoughts? 

Ben, here's my opinion. :) Personally, I think a documentation parser (similar to my Dexter prototype) should be left to being its own tool. From a maintenance standpoint, and a coding standpoint it just makes good sense. At the very least, meshing dflect with a certain style of documentation reduces its usefulness as one may want to use a different doc tool along with dflect. I think dflect could use a good general-purpose attribute hook similar to what NET has. The most D-like way to achieve this would be to use a pragma():
 version(dflect){ pragma(attribute,<expression>); }

The above expression would be crammed into the static ctor() for the corresponding metadata module, and would have to evaluate to a certain type: say an Attribute class or somesuch. - EricAnderton at yahoo
Jul 16 2005
parent "Ben Hinkle" <ben.hinkle gmail.com> writes:
"pragma" <pragma_member pathlink.com> wrote in message 
news:dbbgfc$1uk2$1 digitaldaemon.com...
 In article <dbbcdh$1rv6$1 digitaldaemon.com>, Ben Hinkle says...
A request for input: what (if any) metadata tags (which aren't already in
standard D) would people like to see? I'm thinking of C#'s attribute 
syntax
and Java's annotations. I suppose the tags will have to be in comments in
order to get past the D compiler. Should doxygen/javadoc comments be added
to the metadata? For example I could imagine something very simple where 
any
comment preceding a method gets added to the metadata for that method as a
"description" property. Or it can try to pick apart tags like  doc, 
 author,
 version, etc etc.

thoughts?

Ben, here's my opinion. :) Personally, I think a documentation parser (similar to my Dexter prototype) should be left to being its own tool. From a maintenance standpoint, and a coding standpoint it just makes good sense. At the very least, meshing dflect with a certain style of documentation reduces its usefulness as one may want to use a different doc tool along with dflect.

I agree dflect isn't meant to be used to generate doc. The reason I mentioned the doc string is that dflect is gathering up information about a class for use at run-time. Help is part of that metadata. For example in MATLAB we use Java reflection to let users load jar files and dynamically instantiate and use Java classes from the MATLAB command line. Having short doc strings of what the user just loaded would be very helpful. Otherwise they are on their own to figure out what they have.
 I think dflect could use a good general-purpose attribute hook similar to 
 what
 NET has.  The most D-like way to achieve this would be to use a pragma():

 version(dflect){ pragma(attribute,<expression>); }

The above expression would be crammed into the static ctor() for the corresponding metadata module, and would have to evaluate to a certain type: say an Attribute class or somesuch. - EricAnderton at yahoo

Good idea - though it's a pity the code is so verbose. I'll probably just skip the whole issue for now and dump the parse tree info and that's it.
Jul 16 2005