Digital Mars - D - I Really Need Reflection!

"Craig Black" <cblack ara.com> writes:

Thanks and praise to Walter and all those who are sacrificing their time to
make this innovative programming language a reality!

It sounds like reflection (a.k.a. introspection) is taking a backseat to the
many other features that have been added.  Does anyone realize the power of
class meta-data at run-time?  It's huge!  It's a whole new programming
paradigm!  I could give you many examples of how I would use it if anyone
cared to listen to me rant.

I am very interested in D and all of its innovate features, but for my
purposes it would not be a vast improvement over C++ if reflection were
absent.  Maybe I'm all by myself in saying this, but I would use D if it
included reflection.  I could not justify porting from C++ otherwise.

I would REALLY like to know what Walter intends to do regarding reflection.

Thanks,

Craig

Mar 29 2004

Dave Sieber <dsieber spamnot.sbcglobal.net> writes:

"Craig Black" <cblack ara.com> wrote:

 It sounds like reflection (a.k.a. introspection) is taking a backseat
 to the many other features that have been added.  Does anyone realize
 the power of class meta-data at run-time?  It's huge!  It's a whole
 new programming paradigm!  I could give you many examples of how I
 would use it if anyone cared to listen to me rant.


It wouldn't be ranting to me :-)  I also want reflection, and find the lack 
of it in C++ one of its largest drawbacks (among many :-)  If D were to 
have reflection support, it would be killer.

 I am very interested in D and all of its innovate features, but for my
 purposes it would not be a vast improvement over C++ if reflection
 were absent.  Maybe I'm all by myself in saying this, but I would use
 D if it included reflection.  I could not justify porting from C++
 otherwise. 


I agree, although I think porting to D would still be an improvement over 
C++, and I would consider it even without reflection (no loss, but many 
gains).

My biggest problem now (besides being a D beginner :-) is lack of debugging 
support. This morning I've been toying with Visual Studio, using a makefile 
wrapped in a VS.NET project. It compiles and I can run the app in the 
Visual Studio debugger, and even have it stop on the source line when an 
exception is thrown. But assert's and other errors don't come through too 
well -- it's possible I haven't set it all up correctly yet, or that it 
isn't possible.

 I would REALLY like to know what Walter intends to do regarding
 reflection. 


Me too, but I wouldn't be shocked if Walter replied (or has already) that 
it isn't in his vision for D.  In any case, it would be a major 
undertaking, not something he could tack on. But I think he could take a 
day or two and implement it for us, doncha think? <BG>

-- 
dave

Mar 29 2004

Ilya Minkov <minkov cs.tum.edu> writes:

Throw your thoughts at us! You can be sure some of us read and comment 
on them. Walter said that he may expand the metadata if someone can 
convince him of the merit of it. So far his arguments were, as far as i 
can recall:

- Serialization can already be done using RTTI - you can query the 
fields of a class.
- The only real use he sees is in development enviroments, which may 
just as well parse the code and compile the thunks, because they 
requiere the compiler anyway.
- It would lead to increased program sizes in a much further extent than 
RTTI and would have much less merit. Besides, it would hinder 
whole-program optimization - unused methods cannot be eliminated, 
non-inline versions can not be omitted.

It also seems that current extended RTTI is not part of the spec, 
perhaps other compilers can implement it in an incompatible manner if 
this can increase program performance - for example interleaving fields 
in struct or class arrays may requiere such measure.

When commenting please also recall that there are interfaces, they are 
enough in most cases. And definetely they are *much* more efficient on 
performance! Also some tasks could be taken by an syntax extension 
mechanism (like OpenC++), of which Walter was thinking for future 
versions of D. Just think that when you need more information with 
interfaces, its generation could then be automated by library means, 
only for the parts where this makes sense!

So, i think i have collected the points here which may make this 
discussion much shorter than the previous one on the same topic. ;) Or 
you might also want to wait with it till Version 1 final comes out and 
we would be collecting ideas for the new generation of D. But of course 
we can discuss it now if you like. And yes, D does not justify to port 
existing software from C++, it justifies mostly to write new software 
making it much less effort.

-eye

Craig Black schrieb:
 Thanks and praise to Walter and all those who are sacrificing their time to
 make this innovative programming language a reality!
 
 It sounds like reflection (a.k.a. introspection) is taking a backseat to the
 many other features that have been added.  Does anyone realize the power of
 class meta-data at run-time?  It's huge!  It's a whole new programming
 paradigm!  I could give you many examples of how I would use it if anyone
 cared to listen to me rant.
 
 I am very interested in D and all of its innovate features, but for my
 purposes it would not be a vast improvement over C++ if reflection were
 absent.  Maybe I'm all by myself in saying this, but I would use D if it
 included reflection.  I could not justify porting from C++ otherwise.
 
 I would REALLY like to know what Walter intends to do regarding reflection.
 
 Thanks,
 
 Craig

Mar 29 2004

"Craig Black" <cblack ara.com> writes:

 - Serialization can already be done using RTTI - you can query the
 fields of a class.


You already store the fields of a class?  I didn't know that!  Great!  In
what form are they stored?  What then do you lack?  How about the methods?
If you already have fields, it would seem to me that the methods would be
easy to do also. Further, it seems that it wouldn't be that hard to add a
complete introspection system. :-)

Actually, having access to fields is the main thing.  But the rest would be
nice to have also, and it would seem to me that this would not take that
long to implement.  You could look at C# if you want to see a well-done
reflection system.

 - The only real use he sees is in development enviroments, which may
 just as well parse the code and compile the thunks, because they
 requiere the compiler anyway.


As far as the file size and memory overhead, it should be a compiler option
to include metadata.  That way you could include it only if you need it.

 - It would lead to increased program sizes in a much further extent than
 RTTI and would have much less merit. Besides, it would hinder
 whole-program optimization - unused methods cannot be eliminated,
 non-inline versions can not be omitted.


This is true.  If you include the methods in the metadata then you would not
be able to discard them.  Even so, if it is a compiler option, this overhead
will only be incured if the programmer specifies the option.

Yes, serialization is one of the main uses of reflection. As far as
development environments, I agree with Walter that the DE should be able to
parse the code.  However, there are many other uses for reflection.  As I
said, this is a whole new programming paradigm, the power which has yet to
be discovered.

One cool example of how I would use reflection is for automatic GUI's.  With
reflection, you could automatically generate a GUI based upon the fields
and/or methods in a class.  Just send an instance of your particular class
into the GUI generator, and it will construct a GUI for you to manipulate
the fields, or invoke the methods of that class.  In such a situation, it
would be nice to attach arbitrary information to a particular field/method
so that you could tweak how the GUI behaves.  Correct me if I'm wrong, but I
think that such information is called an "attribute" in C#.

I am confident that there are many more applications for reflection that
have yet to be explored.  It is like any other programming paradigm.  It can
take a while for your brain to think in a new ways.  Thus, you don't realize
the utility of this new approach until you have been using it for a while.

Craig

Mar 29 2004

Dave Sieber <dsieber spamnot.sbcglobal.net> writes:

"Craig Black" <cblack ara.com> wrote:

 You already store the fields of a class?  I didn't know that!  Great! 
 In what form are they stored?  What then do you lack?  How about the
 methods? If you already have fields, it would seem to me that the
 methods would be easy to do also. Further, it seems that it wouldn't
 be that hard to add a complete introspection system. :-)


Hmm, I can't find anywhere in the docs that says information about the 
fields is available at runtime. In fact, looking at ClassInfo in object.d 
in the phobos library, there is information on the base class and the 
interfaces, which would be used for dynamic casting, and a few other 
things, but no information about the methods or fields.

 Actually, having access to fields is the main thing.  But the rest
 would be nice to have also, and it would seem to me that this would
 not take that long to implement.  You could look at C# if you want to
 see a well-done reflection system.


I agree that C# has a very good reflection system. In fact, if C# could be 
compiled to native code (and didn't require the .NET runtime) I would 
probably be using it for most of my development. So I hope it's "D to the 
rescue!" :-)

 As far as the file size and memory overhead, it should be a compiler
 option to include metadata.  That way you could include it only if you
 need it. 


Yes -- following the C++ philosophy that you don't pay for it if you don't 
use it.

-- 
dave

Mar 29 2004

Ilya Minkov <minkov cs.tum.edu> writes:

Dave Sieber schrieb:

 "Craig Black" <cblack ara.com> wrote:


 Hmm, I can't find anywhere in the docs that says information about the 
 fields is available at runtime. In fact, looking at ClassInfo in object.d 
 in the phobos library, there is information on the base class and the 
 interfaces, which would be used for dynamic casting, and a few other 
 things, but no information about the methods or fields.


Good that you looked. I looked a bit through the compiler source and it 
really seems to lack it these days.

Then i'm looking now at the source of DLI (changes from it should be 
easy to reintegrate with DMD), which introduced the extended RTTI, which 
as i now suspect did not make it back into the main distro. It 
apparently at least has placeholders for such things as FieldInfo, 
EnumInfo, StructInfo, MethodInfo. Its version of Phobos includes a 
working module pickle, which saves a tree of classes and arrays and can 
restore it. I tried to compile it on current DMD, but it apparently 
requieres more definitions from its object.d, which incude SymbolInfo, 
huge ClassInfo including FieldInfo[] and MethodInfo[], and all other 
definitions to support it. It would also all requiere compiler 
support... Which i think should be reintegrated.

I'm sorry i deceived you all.

This also makes it more clear why Butron, the author of DLI, has been 
loosing interest in D. He was probably not finding enough support for 
his effort from Walter.

As far as the file size and memory overhead, it should be a compiler
option to include metadata.  That way you could include it only if you
need it. 


 Yes -- following the C++ philosophy that you don't pay for it if you don't 
 use it.


No, it should probably not. It should be a per-class option.

-eye

Mar 31 2004

"Julio C�sar Carrascal Urquijo" <jcesar phreaker.net> writes:

"Craig Black" <cblack ara.com> escribi� en el mensaje
news:c49mja$2tc4$1 digitaldaemon.com...
(...)
 It sounds like reflection (a.k.a. introspection) is taking a backseat to


 many other features that have been added.  Does anyone realize the power


 class meta-data at run-time?  It's huge!  It's a whole new programming
 paradigm!  I could give you many examples of how I would use it if anyone
 cared to listen to me rant.


Agreed. There's support for RTTI and Serialization already but class
meta-data is really handy, especially if you want to build a Form Designer,
Web Services or just loading code from a file.

 I am very interested in D and all of its innovate features, but for my
 purposes it would not be a vast improvement over C++ if reflection were
 absent.  Maybe I'm all by myself in saying this, but I would use D if it
 included reflection.  I could not justify porting from C++ otherwise.


Well, I think reflection, as implemented in Java or .NET (I mean creating
and executing code at runtime), isn't that important for D right now, but
class meta-data is. Also, don't think it's that hard for Walter to do it,
but he probably want D as it is right now for the 1.0 release.

Mar 29 2004

"Craig Black" <cblack ara.com> writes:

 Well, I think reflection, as implemented in Java or .NET (I mean creating
 and executing code at runtime), isn't that important for D right now, but
 class meta-data is. Also, don't think it's that hard for Walter to do it,
 but he probably want D as it is right now for the 1.0 release.


Yes, that's what I mean.  Reflection usually refers to metadata plus the
capability to create code at run-time.  Introspection usually refers to
merely metadata.  I use the term reflection loosely because its the popular
term to use.  Introspection is probably what I should have said.
Introspection is the one that makes my mouth water.  Full reflection is too
much for my little brain to handle at the moment.  I would be VERY pleased
with merely metadata.

Craig

Apr 01 2004

"Derek Parnell" <Derek.Parnell psyc.ward> writes:

On Mon, 29 Mar 2004 11:32:26 -0600 (30/Mar/04 03:32:26 AM)
, Craig Black <cblack ara.com> wrote:

 Thanks and praise to Walter and all those who are sacrificing their time  
 to
 make this innovative programming language a reality!

 It sounds like reflection (a.k.a. introspection) is taking a backseat to  
 the
 many other features that have been added.  Does anyone realize the power  
 of
 class meta-data at run-time?  It's huge!  It's a whole new programming
 paradigm!  I could give you many examples of how I would use it if anyone
 cared to listen to me rant.

 I am very interested in D and all of its innovate features, but for my
 purposes it would not be a vast improvement over C++ if reflection were
 absent.  Maybe I'm all by myself in saying this, but I would use D if it
 included reflection.  I could not justify porting from C++ otherwise.

 I would REALLY like to know what Walter intends to do regarding  
 reflection.


You are preaching to the converted here. Reflection deals with situations  
where the data's meta-data cannot be known at compile-time, but is known  
at run-time. This can occur in utility libraries that are used by a third  
party - such as report writers, form builders, data-converters, etc... And  
ideally we are talking about non-built-in datatypes as well, such as  
classes and structs.

-- 
Derek

Mar 29 2004

Richard Krehbiel <rich kastle.com> writes:

Craig Black wrote:
 It sounds like reflection (a.k.a. introspection) is taking a backseat to the
 many other features that have been added.  Does anyone realize the power of
 class meta-data at run-time?  It's huge!


Do you realize the memory space requirements of the full symbol table 
that it takes to do full introspection at run-time?  It's huge!

:-)

I for one would not support automatic inclusion of full introspection to D.

   It's a whole new programming
 paradigm!


Yes, and it's not the domain of *compiled* *to* *machine* *code* languages.


 I could give you many examples of how I would use it if anyone
 cared to listen to me rant.
 
 I am very interested in D and all of its innovate features, but for my
 purposes it would not be a vast improvement over C++ if reflection were
 absent.  Maybe I'm all by myself in saying this, but I would use D if it
 included reflection.  I could not justify porting from C++ otherwise.
 
 I would REALLY like to know what Walter intends to do regarding reflection.


Frankly, Java's just sitting there, ready for your use where you need 
full introspction and reflection.  Or perhaps Python.  But please don't 
engorge compiled languages with full run-time symbol tables - at least, 
not by default.

(I'd have no arguments if Walter wanted to offer a class definition 
modifier saying "this class is fully introspective," at which point it 
could emit symbol table stuff into the object files.  But please, not 
ALL THE TIME.)

Apr 05 2004

Ilya Minkov <minkov cs.tum.edu> writes:

Richard Krehbiel schrieb:
 I for one would not support automatic inclusion of full introspection to D.


Perhaps automatic for fields (because often the whole hierarchy has to 
be serialized) and non-automatic for methods? Method data would also 
tend to be much larger than field data, i think.

Syntactically, inclusion of such information with classes can be done by 
inheriting them from a "magic" interface which would also be used to 
access it. Actually, even field data can be placed optionally this way 
since it's so simple to use.

   It's a whole new programming paradigm!


 Yes, and it's not the domain of *compiled* *to* *machine* *code* languages.


Which leads me to an idea of companion scripting language to D, which 
should allow easy combination of scripted and native code. But that's a 
whole new direction of work! And hey, it would need near to full 
introspection capabilities in the compiled part of D.

-eye

Apr 05 2004

"Walter" <walter digitalmars.com> writes:

"Craig Black" <cblack ara.com> wrote in message
news:c49mja$2tc4$1 digitaldaemon.com...
 I would REALLY like to know what Walter intends to do regarding



I do intend to expand the 'classinfo' so that one could figure out all the
fields and methods of a class. However, I spent some time trying to figure
out how to make dynamic class loading work with some special compiler magic,
but it isn't going to happen. The stumbling block is not knowing at compile
time what the type of an expression is.

I could wave the magic spec wand and say that all methods in a dynamic class
must return void and that data fields are not possible, but such
restrictions make it not so useful.

Apr 08 2004

J Anderson <REMOVEanderson badmama.com.au> writes:

Walter wrote:

"Craig Black" <cblack ara.com> wrote in message
news:c49mja$2tc4$1 digitaldaemon.com...
  

I would REALLY like to know what Walter intends to do regarding
    



I do intend to expand the 'classinfo' so that one could figure out all the
fields and methods of a class. However, I spent some time trying to figure
out how to make dynamic class loading work with some special compiler magic,
but it isn't going to happen. The stumbling block is not knowing at compile
time what the type of an expression is.

I could wave the magic spec wand and say that all methods in a dynamic class
must return void and that data fields are not possible, but such
restrictions make it not so useful.
  


section in the wiki.

http://www.prowiki.org/wiki4d/wiki.cgi?HowTo/RealtimeTypeInformation

-- 
-Anderson: http://badmama.com.au/~anderson/

Apr 08 2004

"Craig Black" <cblack ara.com> writes:

 I do intend to expand the 'classinfo' so that one could figure out all the
 fields and methods of a class.


Super-de-duper! What version do you forsee as having these capabilities?

However, I spent some time trying to figure
 out how to make dynamic class loading work with some special compiler


 but it isn't going to happen. The stumbling block is not knowing at


 time what the type of an expression is.


"Dynamic class loading"?  You're talking about loading a class from a DLLat
run-time, right?  I don't really understand your problem.  Perhaps I have a
different idea about what "dynamic class loading" is.

 I could wave the magic spec wand and say that all methods in a dynamic


 must return void and that data fields are not possible, but such
 restrictions make it not so useful.


Again I don't understand your dilemma.  Perhaps, if you have time, you could
expound on it.

Thanks,

Craig

Apr 09 2004

"Walter" <newshound digitalmars.com> writes:

"Craig Black" <cblack ara.com> wrote in message
news:c572ms$j7c$1 digitaldaemon.com...
 I do intend to expand the 'classinfo' so that one could figure out all




 fields and methods of a class.


 Super-de-duper! What version do you forsee as having these capabilities?


2.0.

However, I spent some time trying to figure
 out how to make dynamic class loading work with some special compiler


 but it isn't going to happen. The stumbling block is not knowing at


 time what the type of an expression is.




 run-time, right?  I don't really understand your problem.  Perhaps I have


 different idea about what "dynamic class loading" is.


I mean given the name of a class, load it at runtime from some external
database (like a DLL), create an instance of that class, and then access its
members.

 I could wave the magic spec wand and say that all methods in a dynamic


 must return void and that data fields are not possible, but such
 restrictions make it not so useful.


 Again I don't understand your dilemma.  Perhaps, if you have time, you


 expound on it.


Object o = classFactory("TheNameOfSomeClassNotKnownAtCompileTime");
int a = o.x + o.y;

At runtime we can determine if the fields x and y are present, and what
their types are. But we don't know how to compile the + expression, since we
don't know the types of x and y at compile time. Think of how this would
impact function overloading, for example:

    foo(o.x);

Aug 09 2004

teqDruid <me teqdruid.com> writes:

On Mon, 09 Aug 2004 16:15:03 -0700, Walter wrote:

 
 "Craig Black" <cblack ara.com> wrote in message
 news:c572ms$j7c$1 digitaldaemon.com...
 I do intend to expand the 'classinfo' so that one could figure out all




 fields and methods of a class.


 Super-de-duper! What version do you forsee as having these capabilities?


 2.0.
 
However, I spent some time trying to figure
 out how to make dynamic class loading work with some special compiler


 but it isn't going to happen. The stumbling block is not knowing at


 time what the type of an expression is.




 run-time, right?  I don't really understand your problem.  Perhaps I have


 different idea about what "dynamic class loading" is.


 I mean given the name of a class, load it at runtime from some external
 database (like a DLL), create an instance of that class, and then access its
 members.
 
 I could wave the magic spec wand and say that all methods in a dynamic


 must return void and that data fields are not possible, but such
 restrictions make it not so useful.


 Again I don't understand your dilemma.  Perhaps, if you have time, you


 expound on it.


 Object o = classFactory("TheNameOfSomeClassNotKnownAtCompileTime");
 int a = o.x + o.y;


Something like this isn't a problem with classes, but rather a typing
issue, and (obviously) the way to resolve it is using a cast.  For
instance if you were to provide a classFactory method, one would then do:

Object o = classFactory("TheNameOfSomeClassNotKnownAtCompileTime");
MyInterface a = cast(MyInterface)o;
if (a is null)
	throw new Exception("Bad Class");

Casting is probably the most consistent method of doing anything like
this- you even have to cast with Reflection in Java- and I really like the
way they do it.

For those of you who aren't familiar, in Java for each class you write,
there is a Class class as well.  That is to say, one can get a Class
object representing your class.  One of that Object's fields is an array
of Constructor objects (one can even do a Class.getConstructor(Class[]
parameterTypes) which was a HUGE pain before Java got autoboxing in 1.5).
Once one has a Constructor, run this on the constructor object:
"Object newInstance(Object[] initargs) " <- right out of JDocs.

One can also do similar things with Method objects in Java.

I'm sure that Walter has thought of simply casting, but has a better
method (as he frequently does) although I'm having trouble imagining any
other method that is type-safe.  So the question is... what'cha got
brewing up there, Walter?

John

Aug 10 2004

School <school users.sf.net> writes:

Walter ����:

 "Craig Black" <cblack ara.com> wrote in message
 news:c49mja$2tc4$1 digitaldaemon.com...
 
I would REALLY like to know what Walter intends to do regarding


 reflection.
 
 I do intend to expand the 'classinfo' so that one could figure out all the
 fields and methods of a class. However, I spent some time trying to figure
 out how to make dynamic class loading work with some special compiler magic,
 but it isn't going to happen. The stumbling block is not knowing at compile
 time what the type of an expression is.
 
 I could wave the magic spec wand and say that all methods in a dynamic class
 must return void and that data fields are not possible, but such
 restrictions make it not so useful.
 
 


Check it   http://reflection.sourceforge.net/
-- 
School, yet another nickname for anonymous.
:D ;-D

Apr 11 2004