www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - TypeInfoEx and Variant: suggestions?

reply Christopher Wright <dhasenan gmail.com> writes:
Hi all,

I finally found some time to work on TypeInfoEx. It's usable, in that 
you can view information about a type; but you can't do anything with 
that information. One large barrier is the lack of a usable Variant type.

To invoke a function on an object or struct, you need to pass it as 
either Object or void* or Variant.

If you invoke a function, you want to pass in an array of Variant for 
the arguments. You want to get the return type as a Variant.

If you get a field, you want to get it as a Variant.

You could use templates instead, but that's not usable in many 
situations, so I won't have that as the only option.

Andrei's std.variant has an interesting design, but it has two flaws 
that render it unusable:
  - It cannot be used with structs of arbitrary size. It's parameterized 
on size, but no size will always be sufficient; and due to its 
allocation scheme, choosing to support (for instance) 1KB structs will 
force a bool to take 1KB.
  - It cannot be created with RTTI; it requires compile-time type 
information.

So, my options are:
  - Write my own Variant supporting the necessary operations. This makes 
things annoying for anyone wanting to use my code with Phobos, since now 
there are two Variant structs running around.
  - Use std.boxer, which is deprecated.
  - Pass around an opaque Pair!(TypeInfo, void*).

Any suggestions? I've written such a Variant for D1 and Tango, but I 
relish neither the idea of supporting both with the same class, nor code 
duplication.
Apr 16 2009
next sibling parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Christopher Wright wrote:
 Hi all,
 
 I finally found some time to work on TypeInfoEx. It's usable, in that
 you can view information about a type; but you can't do anything with
 that information. One large barrier is the lack of a usable Variant type.
 
 To invoke a function on an object or struct, you need to pass it as
 either Object or void* or Variant.
 
 If you invoke a function, you want to pass in an array of Variant for
 the arguments. You want to get the return type as a Variant.
 
 If you get a field, you want to get it as a Variant.
 
 You could use templates instead, but that's not usable in many
 situations, so I won't have that as the only option.
 
 Andrei's std.variant has an interesting design, but it has two flaws
 that render it unusable:
  - It cannot be used with structs of arbitrary size. It's parameterized
 on size, but no size will always be sufficient; and due to its
 allocation scheme, choosing to support (for instance) 1KB structs will
 force a bool to take 1KB.
  - It cannot be created with RTTI; it requires compile-time type
 information.
 
 So, my options are:
  - Write my own Variant supporting the necessary operations. This makes
 things annoying for anyone wanting to use my code with Phobos, since now
 there are two Variant structs running around.
  - Use std.boxer, which is deprecated.
  - Pass around an opaque Pair!(TypeInfo, void*).
 
 Any suggestions? I've written such a Variant for D1 and Tango, but I
 relish neither the idea of supporting both with the same class, nor code
 duplication.

Use the Tango Variant code. It supports values of any size and doesn't require knowledge of the compile-time type. It's in tango.core so the dependencies are relatively small, so porting should be fairly simple (the code originally compiled under both Phobos and Tango, but that version's out of date now.) If you look at http://dsource.org/projects/tango/ticket/821 there's a patch to the current Variant that adds support for varargs as well. -- Daniel
Apr 16 2009
next sibling parent reply Christopher Wright <dhasenan gmail.com> writes:
Daniel Keep wrote:
 Use the Tango Variant code.  It supports values of any size and doesn't
 require knowledge of the compile-time type.

Not so. That was one of three reasons that I wrote my own Variant previously (the others being toString and sameness comparison).
Apr 16 2009
parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Christopher Wright wrote:
 Daniel Keep wrote:
 Use the Tango Variant code.  It supports values of any size and doesn't
 require knowledge of the compile-time type.

Not so. That was one of three reasons that I wrote my own Variant previously (the others being toString and sameness comparison).

What not so? I know for a fact it supports values of any size, so I assume you're referring to compile-time. With the new patch you can pack and unpack Variants from TypeInfo and a void* so you DON'T need to know the type ahead of time. There isn't a way to construct from a single TypeInfo,void* pair since I wasn't sure if that would actually be needed. toString is a design issue, not a technical one; they didn't want stuff in tango.core depending on stuff outside of that. If you look at the history of Variant, there's an old version with "proper" toString code that should port forward painlessly. On a side note, I'm working on getting Tango's Layout to understand Variants and do the "proper" stringification. Still trying to work out how to get the changes to compile without having to rebuild Tango... *grumble* As for comparisons, it supports opEqual and opCmp between Variants, it just doesn't attempt to do any automatic conversion between non-identical but comparable types (like comparing ints to longs.) Is there something specific you think Tango's Variant should do that it doesn't? -- Daniel
Apr 16 2009
next sibling parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Steven Schveighoffer wrote:
 On Thu, 16 Apr 2009 21:37:29 -0400, Daniel Keep
 <daniel.keep.lists gmail.com> wrote:
 
 On a side note, I'm working on getting Tango's Layout to understand
 Variants and do the "proper" stringification.  Still trying to work out
 how to get the changes to compile without having to rebuild Tango...
 *grumble*

Have you tried building your test app with your modified Tango file specifically? That generally works for me: dmd variantTest.d ~/d/include/d/tango/core/Variant.d -Steve

That works for Variant, but it doesn't seem to work with Layout. It compiles and runs, but it doesn't seem to actually use the code I give it. -- Daniel
Apr 17 2009
parent Daniel Keep <daniel.keep.lists gmail.com> writes:
Steven Schveighoffer wrote:
 On Fri, 17 Apr 2009 07:11:33 -0400, Daniel Keep
 <daniel.keep.lists gmail.com> wrote:
 
 Steven Schveighoffer wrote:
 On Thu, 16 Apr 2009 21:37:29 -0400, Daniel Keep
 <daniel.keep.lists gmail.com> wrote:

 On a side note, I'm working on getting Tango's Layout to understand
 Variants and do the "proper" stringification.  Still trying to work out
 how to get the changes to compile without having to rebuild Tango...
 *grumble*

Have you tried building your test app with your modified Tango file specifically? That generally works for me: dmd variantTest.d ~/d/include/d/tango/core/Variant.d -Steve

That works for Variant, but it doesn't seem to work with Layout. It compiles and runs, but it doesn't seem to actually use the code I give it.

I think I ran into this before. I think it's because Layout is a template, and is compiled when it is used, not when the module is compiled. You have to recompile the user of Layout as well (i.e. Stdout.d, and probably Print.d). -Steve

Ah, but I'm using it directly. AND I've put my main function inside the modified Layout module. What more does it want from me? Cookies? Chocolates?! -- Daniel
Apr 17 2009
prev sibling parent Christopher Wright <dhasenan gmail.com> writes:
Daniel Keep wrote:
 
 Christopher Wright wrote:
 Daniel Keep wrote:
 Use the Tango Variant code.  It supports values of any size and doesn't
 require knowledge of the compile-time type.

previously (the others being toString and sameness comparison).

What not so? I know for a fact it supports values of any size, so I assume you're referring to compile-time. With the new patch you can pack and unpack Variants from TypeInfo and a void* so you DON'T need to know the type ahead of time. There isn't a way to construct from a single TypeInfo,void* pair since I wasn't sure if that would actually be needed.

Okay, you must have done that very recently, since I can't find it in my working copy.
 toString is a design issue, not a technical one; they didn't want stuff
 in tango.core depending on stuff outside of that.  If you look at the
 history of Variant, there's an old version with "proper" toString code
 that should port forward painlessly.

Yes, I'm familiar with that issue. I'm not faulting Tango for lacking a proper toString, but it was a problem, one that I couldn't work around with Tango's Variant, so I did what I had to do.
 On a side note, I'm working on getting Tango's Layout to understand
 Variants and do the "proper" stringification.  Still trying to work out
 how to get the changes to compile without having to rebuild Tango...
 *grumble*
 
 As for comparisons, it supports opEqual and opCmp between Variants, it
 just doesn't attempt to do any automatic conversion between
 non-identical but comparable types (like comparing ints to longs.)

Ah. Mine does that -- it stores all floating point types as reals, for instance, and all integer types as longs (except for ulongs).
 Is there something specific you think Tango's Variant should do that it
 doesn't?

Apparently, nothing that won't be in the next release, aside from bitwise comparisons.
   -- Daniel

Apr 17 2009
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 16 Apr 2009 21:37:29 -0400, Daniel Keep  
<daniel.keep.lists gmail.com> wrote:

 On a side note, I'm working on getting Tango's Layout to understand
 Variants and do the "proper" stringification.  Still trying to work out
 how to get the changes to compile without having to rebuild Tango...
 *grumble*

Have you tried building your test app with your modified Tango file specifically? That generally works for me: dmd variantTest.d ~/d/include/d/tango/core/Variant.d -Steve
Apr 17 2009
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 17 Apr 2009 07:11:33 -0400, Daniel Keep  
<daniel.keep.lists gmail.com> wrote:

 Steven Schveighoffer wrote:
 On Thu, 16 Apr 2009 21:37:29 -0400, Daniel Keep
 <daniel.keep.lists gmail.com> wrote:

 On a side note, I'm working on getting Tango's Layout to understand
 Variants and do the "proper" stringification.  Still trying to work out
 how to get the changes to compile without having to rebuild Tango...
 *grumble*

Have you tried building your test app with your modified Tango file specifically? That generally works for me: dmd variantTest.d ~/d/include/d/tango/core/Variant.d -Steve

That works for Variant, but it doesn't seem to work with Layout. It compiles and runs, but it doesn't seem to actually use the code I give it.

I think I ran into this before. I think it's because Layout is a template, and is compiled when it is used, not when the module is compiled. You have to recompile the user of Layout as well (i.e. Stdout.d, and probably Print.d). -Steve
Apr 17 2009
prev sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 17 Apr 2009 07:23:48 -0400, Daniel Keep  
<daniel.keep.lists gmail.com> wrote:

 Steven Schveighoffer wrote:
 On Fri, 17 Apr 2009 07:11:33 -0400, Daniel Keep
 <daniel.keep.lists gmail.com> wrote:

 Steven Schveighoffer wrote:
 On Thu, 16 Apr 2009 21:37:29 -0400, Daniel Keep
 <daniel.keep.lists gmail.com> wrote:

 On a side note, I'm working on getting Tango's Layout to understand
 Variants and do the "proper" stringification.  Still trying to work  
 out
 how to get the changes to compile without having to rebuild Tango...
 *grumble*

Have you tried building your test app with your modified Tango file specifically? That generally works for me: dmd variantTest.d ~/d/include/d/tango/core/Variant.d -Steve

That works for Variant, but it doesn't seem to work with Layout. It compiles and runs, but it doesn't seem to actually use the code I give it.

I think I ran into this before. I think it's because Layout is a template, and is compiled when it is used, not when the module is compiled. You have to recompile the user of Layout as well (i.e. Stdout.d, and probably Print.d). -Steve

Ah, but I'm using it directly. AND I've put my main function inside the modified Layout module. What more does it want from me? Cookies? Chocolates?!

You must be doing something wrong, it works for me: [steves localhost trunk]$ svn diff tango/text/convert/Layout.d Index: tango/text/convert/Layout.d =================================================================== --- tango/text/convert/Layout.d (revision 4435) +++ tango/text/convert/Layout.d (working copy) -35,6 +35,8 private import tango.io.model.IConduit : OutputStream; +private import tango.stdc.stdio; + /******************************************************************************* Platform issues ... -602,6 +604,7 return integer (result, *cast(ushort*) p, format, ushort.max, "u"); case TypeCode.INT: + printf("inside INT print!\n"); return integer (result, *cast(int*) p, format, uint.max); case TypeCode.UINT: [steves localhost trunk]$ cd ~/dstuff/testing [steves localhost testing]$ cat teststuff.d import tango.text.convert.Layout; void main() { auto layout = new Layout!(char); int x = 0; auto result = layout("{}", x); } [steves localhost testing]$ dmd teststuff.d [steves localhost testing]$ ./teststuff inside INT print! [steves localhost testing]$ -Steve
Apr 17 2009
prev sibling parent reply dsimcha <dsimcha yahoo.com> writes:
== Quote from Christopher Wright (dhasenan gmail.com)'s article
 Hi all,
 I finally found some time to work on TypeInfoEx. It's usable, in that
 you can view information about a type; but you can't do anything with
 that information. One large barrier is the lack of a usable Variant type.
 To invoke a function on an object or struct, you need to pass it as
 either Object or void* or Variant.
 If you invoke a function, you want to pass in an array of Variant for
 the arguments. You want to get the return type as a Variant.
 If you get a field, you want to get it as a Variant.
 You could use templates instead, but that's not usable in many
 situations, so I won't have that as the only option.
 Andrei's std.variant has an interesting design, but it has two flaws
 that render it unusable:
   - It cannot be used with structs of arbitrary size. It's parameterized
 on size, but no size will always be sufficient; and due to its
 allocation scheme, choosing to support (for instance) 1KB structs will
 force a bool to take 1KB.
   - It cannot be created with RTTI; it requires compile-time type
 information.
 So, my options are:
   - Write my own Variant supporting the necessary operations. This makes
 things annoying for anyone wanting to use my code with Phobos, since now
 there are two Variant structs running around.
   - Use std.boxer, which is deprecated.
   - Pass around an opaque Pair!(TypeInfo, void*).
 Any suggestions? I've written such a Variant for D1 and Tango, but I
 relish neither the idea of supporting both with the same class, nor code
 duplication.

Andrei has said that he would like to fix std.variant to support arbitrary sized stuff, but just doesn't have time right now. If you are able and willing, submit a patch. If it's decent, it will probably be accepted. If you don't have time or don't want to, someone who does should submit one since this seems to be something a lot of people want. I've looked at doing this before, and I'd be willing to step up and submit a patch after some of my coursework dies down and I have more free time (fairly soon), if noone else gets around to it before then.
Apr 16 2009
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
dsimcha wrote:
 == Quote from Christopher Wright (dhasenan gmail.com)'s article
 Hi all,
 I finally found some time to work on TypeInfoEx. It's usable, in that
 you can view information about a type; but you can't do anything with
 that information. One large barrier is the lack of a usable Variant type.
 To invoke a function on an object or struct, you need to pass it as
 either Object or void* or Variant.
 If you invoke a function, you want to pass in an array of Variant for
 the arguments. You want to get the return type as a Variant.
 If you get a field, you want to get it as a Variant.
 You could use templates instead, but that's not usable in many
 situations, so I won't have that as the only option.
 Andrei's std.variant has an interesting design, but it has two flaws
 that render it unusable:
   - It cannot be used with structs of arbitrary size. It's parameterized
 on size, but no size will always be sufficient; and due to its
 allocation scheme, choosing to support (for instance) 1KB structs will
 force a bool to take 1KB.
   - It cannot be created with RTTI; it requires compile-time type
 information.
 So, my options are:
   - Write my own Variant supporting the necessary operations. This makes
 things annoying for anyone wanting to use my code with Phobos, since now
 there are two Variant structs running around.
   - Use std.boxer, which is deprecated.
   - Pass around an opaque Pair!(TypeInfo, void*).
 Any suggestions? I've written such a Variant for D1 and Tango, but I
 relish neither the idea of supporting both with the same class, nor code
 duplication.

Andrei has said that he would like to fix std.variant to support arbitrary sized stuff, but just doesn't have time right now. If you are able and willing, submit a patch. If it's decent, it will probably be accepted. If you don't have time or don't want to, someone who does should submit one since this seems to be something a lot of people want. I've looked at doing this before, and I'd be willing to step up and submit a patch after some of my coursework dies down and I have more free time (fairly soon), if noone else gets around to it before then.

It would be great if somebody could find the time to implement the pesky arbitrarily-sized structs for Variant. I don't have much time to add to this interesting discussion, but let me make a few quick points. First off, I'm very happy that discriminated unions are receiving increased interest from the community. I think they are an interesting abstraction mechanism with cool applications in dynamic code. Second, I have hacked discriminated unions for years and wrote a fairly influential paper on them as far back as 2001 (which has inspired many others, including Boost's), and some more articles later on. I can say that std.variant is the best implementation of a discriminated I've seen, by a mile. std.variant is truly general (at no point in its design or implementation appears a list of supported types), well positioned for working with full introspection when that will become available, tight in memory usage, and extremely fast because it uses only one indirect call for dispatching. The latter point is very important in heavy-duty usage; hash-based dispatch sounds and looks good on paper, until you need to invoke it in a tight loop. Then you'd start trying to avoid it, which undermines motivation for using variants. Lack of construction from RTTI is not a major limitation and can be obviated. RTTI objects cannot be forged, so they were created from some objects that did have static type information available. This suggests that code can be shuffled such that Variant construction is done at that point, instead of getting the RTTI alone (and losing static type information) and attempting to construct the Variant later. Andrei
Apr 16 2009