www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Better tuples

reply bearophile <bearophileHUGS lycos.com> writes:
I'd like to add five enhancement requests in Bugzilla, related to
std.typecons.Tuple. I show them here first for possible comments or critiques.

==========================================

Title: [Better tuples] Better tuples

(In this bug report I talk about something similar to std.typecons.Tuple. A
problem with the "tuple" name: structs have a "tupleof" field, but it returns
something quite different from std.typecons.Tuple. This name clash must be
addressed somehow to avoid confusion of newbie D programmers.)

Some languages as Python show that good tuples (inhomogeneous sequences of
values of arbitrary types, in D they are statically typed) are very handy, they
help a lot in high-level coding.

Some syntactic support can make tuple usage natural, easy and widespread in D
programs written by all people, even newbies. But the lack of such syntax can
keep them a library feature used only by few programmers and makes impossible
some very useful tuples usage patterns. From my experience I think that in a
language as D that allows operator overloading built-in tuples are more useful
than built-in associative arrays (the main advantage of built-in associative
arrays are their type literals that enjoy type inference).

I have seen that std.typecons.Tuple is quite useful, but it misses some very
important features. Some features can just be added to it (see for example bug
4381 ) and maybe the apply(). But other features can't just be added, they need
some syntax and semantic support.

Some other useful things:
- More integration and usage of Tuple in druntime. For example the associative
array property AA.byItem() can yield tuple(key,value) lazily, and AA.items can
return a dynamic array of them.
- Some way to reverse the order of the items of a tuple (generating a tuple of
different type).
- "in" operator for tuples (as for arrays).
- More tuple-aware functions in Phobos, like a function to build an AA from a
range of tuple(key, value).
- Optional: Zip and other pairing ranges to yield tuple (currently for
efficiency they yield a pair of pointers. But this breaks abstraction. Using a
more common data structure has significant advantages. The compiler can
recognize the idiom and in some cases can avoid the creation of the Tuples that
Zip produce.

------------

See the enhancement requests: bug 4381 , bug xxxx , bug xxxx , bug xxxx , bug
xxxx , bug xxxx

(This bug report will have dependencies on other six but reports.)
(I will append to bug 4381 a reference to this bug.)

==========================================

Title: [Better tuples] Tuple unpacking

This is one of the tuple enhancement requests. See bug xxxx for an introduction.

Tuple unpacking is quite handy (the following syntax is just an syntax-idea,
other syntaxes can be used. This syntax is not valid in C, so I think this
syntax can be used in D):

auto foo() {
    return tuple(10, "hello");
}
void main() {
    (int n, string s) = foo();
    writeln(n, " ", s);
    (n, s) = foo(); // calls is again
    int[] arr = [1, 2, 3];
    (int a, int b, int c) = arr; // other kind of unpacking
    int2[] arr2 = tuple(1, 2, 3);
    (auto n2, auto s2) = foo(); // calls is again
}

------------

This is the syntax that can be currently used:

auto foo() {
    T1 alpha = computeIt1(...);
    T1 alpha = computeIt2(...);
    return Tuple!(T1, "alpha", T2, "beta")(alpha, beta);
    //return tuple(alpha, beta); // alternative
}
void main() {
    auto alpha_beta = foo();
    use1(alpha_beta.alpha);
    use2(alpha_beta.beta);
    use1(alpha_beta.field[0]); // alternative
    use2(alpha_beta.field[1]);  // alternative    
}

------------

More syntax that can be currently used:

import std.stdio, std.typecons;
void main() {
    int[] arr2 = [tuple(1, 2, 3).tupleof];
    writeln(arr2);
}

But it prints:
1 2 3 1 2 3
Instead of:
[1, 2, 3]

------------

Optional features, more advanced, that can be added later if they are seen
worth it. They are inspired by Python 2.x and 3.x syntax.

Nested unpacking:

auto foo() {
  return tuple(10, tuple("hello", 1.5));
}
void main() {
  (int i, (string s, double d)) = foo();
}



Python 3.x supports a partial tuple unpacking (here 'rest' will contain the
tuple ("hello", 1.5) ):

def bar():
  return (10, "hello", 1.5)
i, *rest = foo()

But I don't know if this is worth adding to D.

==========================================

Title: [Better tuples] [] syntax support for tuples

This is one of the tuple enhancement requests. See bug xxxx for an introduction.

opIndex/opIndexAssign syntax support for tuples:

auto tup = tuple(10, 20, 30, 40);
writeln(tup[0]);
tup[1] = 5;
int y = tup[2];
tup[3]++;


Slice syntax for tuples (currently done with 'Tuple.slice'):
auto tup = tuple(10, 20, 30, 40, 50, 60);
auto part = tup[1 .. 3];


The type of tuple elements can differ, so the index must be known at
compile-time, just as with tupleof[].

A possible way to implement it (currently this can't be used):
alias this.tupleof[0..$] this;

==========================================

Title: [Better tuples] (static) foreach on tuple items

This is one of the tuple enhancement requests. See bug xxxx for an introduction.

auto tup = tuple(10, 20, 30);
static foreach (item; tup)
    writeln(item);


See also bug 4085


Note: in dmd v2.047 this prints all fields twice:

foreach (item; tup.tupleof)
    writeln(item);

==========================================

Title: [Better tuples] Apply with tuples

This is one of the tuple enhancement requests. See bug xxxx for an introduction.

Apply functionality is quite useful, as shown by this small Python program that
prints "1, 2":

def foo(x, y):
    print x, y
tuple1 = (1, 2)
foo(*tuple1)


That star syntax of Python is equivalent to this, that works still in Python
2.x:

def foo(x, y):
    print x, y
tuple1 = (1, 2)
apply(foo, tuple1)

The star syntax can't be used in D, but a good apply() function can be useful
in D too.


A possible usage in D:

void foo(T1, T2)(T1 x, T2 y) {
    writeln(x, " ", y);
}
void main() {
    auto tuple1 = tuple(1, 2);
    apply(&foo!(tuple1[0].typeof, tuple1[1].typeof), tuple1);
    auto tuple2 = tuple(1, 2, 3);
    apply(&foo!(tuple2[0].typeof, tuple2[1].typeof), tuple2[0..2]);
}

==========================================

Bye,
bearophile
Jun 30 2010
next sibling parent reply Eric Poggel <dnewsgroup yage3d.net> writes:
On 6/30/2010 7:13 PM, bearophile wrote:

 I'd like to add five enhancement requests in Bugzilla, related to
std.typecons.Tuple. I show them here first for possible comments or critiques.

 ==========================================

 Title: [Better tuples] Better tuples

 (In this bug report I talk about something similar to std.typecons.Tuple. A
problem with the "tuple" name: structs have a "tupleof" field, but it returns
something quite different from std.typecons.Tuple. This name clash must be
addressed somehow to avoid confusion of newbie D programmers.)

 Some languages as Python show that good tuples (inhomogeneous sequences of
values of arbitrary types, in D they are statically typed) are very handy, they
help a lot in high-level coding.

 Some syntactic support can make tuple usage natural, easy and widespread in D
programs written by all people, even newbies. But the lack of such syntax can
keep them a library feature used only by few programmers and makes impossible
some very useful tuples usage patterns. From my experience I think that in a
language as D that allows operator overloading built-in tuples are more useful
than built-in associative arrays (the main advantage of built-in associative
arrays are their type literals that enjoy type inference).

 I have seen that std.typecons.Tuple is quite useful, but it misses some very
important features. Some features can just be added to it (see for example bug
4381 ) and maybe the apply(). But other features can't just be added, they need
some syntax and semantic support.

 Some other useful things:
 - More integration and usage of Tuple in druntime. For example the associative
array property AA.byItem() can yield tuple(key,value) lazily, and AA.items can
return a dynamic array of them.
 - Some way to reverse the order of the items of a tuple (generating a tuple of
different type).
 - "in" operator for tuples (as for arrays).
 - More tuple-aware functions in Phobos, like a function to build an AA from a
range of tuple(key, value).
 - Optional: Zip and other pairing ranges to yield tuple (currently for
efficiency they yield a pair of pointers. But this breaks abstraction. Using a
more common data structure has significant advantages. The compiler can
recognize the idiom and in some cases can avoid the creation of the Tuples that
Zip produce.

 ------------

 See the enhancement requests: bug 4381 , bug xxxx , bug xxxx , bug xxxx , bug
xxxx , bug xxxx

 (This bug report will have dependencies on other six but reports.)
 (I will append to bug 4381 a reference to this bug.)

 ==========================================

 Title: [Better tuples] Tuple unpacking

 This is one of the tuple enhancement requests. See bug xxxx for an
introduction.

 Tuple unpacking is quite handy (the following syntax is just an syntax-idea,
other syntaxes can be used. This syntax is not valid in C, so I think this
syntax can be used in D):

 auto foo() {
      return tuple(10, "hello");
 }
 void main() {
      (int n, string s) = foo();
      writeln(n, " ", s);
      (n, s) = foo(); // calls is again
      int[] arr = [1, 2, 3];
      (int a, int b, int c) = arr; // other kind of unpacking
      int2[] arr2 = tuple(1, 2, 3);
      (auto n2, auto s2) = foo(); // calls is again
 }

 ------------

 This is the syntax that can be currently used:

 auto foo() {
      T1 alpha = computeIt1(...);
      T1 alpha = computeIt2(...);
      return Tuple!(T1, "alpha", T2, "beta")(alpha, beta);
      //return tuple(alpha, beta); // alternative
 }
 void main() {
      auto alpha_beta = foo();
      use1(alpha_beta.alpha);
      use2(alpha_beta.beta);
      use1(alpha_beta.field[0]); // alternative
      use2(alpha_beta.field[1]);  // alternative
 }

 ------------

 More syntax that can be currently used:

 import std.stdio, std.typecons;
 void main() {
      int[] arr2 = [tuple(1, 2, 3).tupleof];
      writeln(arr2);
 }

 But it prints:
 1 2 3 1 2 3
 Instead of:
 [1, 2, 3]

 ------------

 Optional features, more advanced, that can be added later if they are seen
worth it. They are inspired by Python 2.x and 3.x syntax.

 Nested unpacking:

 auto foo() {
    return tuple(10, tuple("hello", 1.5));
 }
 void main() {
    (int i, (string s, double d)) = foo();
 }



 Python 3.x supports a partial tuple unpacking (here 'rest' will contain the
tuple ("hello", 1.5) ):

 def bar():
    return (10, "hello", 1.5)
 i, *rest = foo()

 But I don't know if this is worth adding to D.

 ==========================================

 Title: [Better tuples] [] syntax support for tuples

 This is one of the tuple enhancement requests. See bug xxxx for an
introduction.

 opIndex/opIndexAssign syntax support for tuples:

 auto tup = tuple(10, 20, 30, 40);
 writeln(tup[0]);
 tup[1] = 5;
 int y = tup[2];
 tup[3]++;


 Slice syntax for tuples (currently done with 'Tuple.slice'):
 auto tup = tuple(10, 20, 30, 40, 50, 60);
 auto part = tup[1 .. 3];


 The type of tuple elements can differ, so the index must be known at
compile-time, just as with tupleof[].

 A possible way to implement it (currently this can't be used):
 alias this.tupleof[0..$] this;

 ==========================================

 Title: [Better tuples] (static) foreach on tuple items

 This is one of the tuple enhancement requests. See bug xxxx for an
introduction.

 auto tup = tuple(10, 20, 30);
 static foreach (item; tup)
      writeln(item);


 See also bug 4085


 Note: in dmd v2.047 this prints all fields twice:

 foreach (item; tup.tupleof)
      writeln(item);

 ==========================================

 Title: [Better tuples] Apply with tuples

 This is one of the tuple enhancement requests. See bug xxxx for an
introduction.

 Apply functionality is quite useful, as shown by this small Python program
that prints "1, 2":

 def foo(x, y):
      print x, y
 tuple1 = (1, 2)
 foo(*tuple1)


 That star syntax of Python is equivalent to this, that works still in Python
2.x:

 def foo(x, y):
      print x, y
 tuple1 = (1, 2)
 apply(foo, tuple1)

 The star syntax can't be used in D, but a good apply() function can be useful
in D too.


 A possible usage in D:

 void foo(T1, T2)(T1 x, T2 y) {
      writeln(x, " ", y);
 }
 void main() {
      auto tuple1 = tuple(1, 2);
      apply(&foo!(tuple1[0].typeof, tuple1[1].typeof), tuple1);
      auto tuple2 = tuple(1, 2, 3);
      apply(&foo!(tuple2[0].typeof, tuple2[1].typeof), tuple2[0..2]);
 }

 ==========================================

 Bye,
 bearophile


I admit that my experience with Tuples is limited, and I haven't put 
much thought into this suggestion, but would it make sense to make 
struct instances and tuples the same thing?  I wonder if there are any 
operations that would make sense on one but not the other?
Jun 30 2010
parent reply bearophile <bearophileHUGS lycos.com> writes:
Eric Poggel:

would it make sense to make struct instances and tuples the same thing?<


It's a nice idea, but if all structs become tuples, and D needs to support
separate compilation, then modules that define structs need to contain the
compiled methods (instantiated templates) that implement all the features of
Tuples. So to keep programs small I think it's better to keep structs simple,
and define a Tuple with richer semantics.

Bye,
bearophile
Jun 30 2010
parent Eric Poggel <dnewsgroup yage3d.net> writes:
On 6/30/2010 8:03 PM, bearophile wrote:

 Eric Poggel:

 would it make sense to make struct instances and tuples the same thing?<


 It's a nice idea, but if all structs become tuples, and D needs to support
separate compilation, then modules that define structs need to contain the
compiled methods (instantiated templates) that implement all the features of
Tuples. So to keep programs small I think it's better to keep structs simple,
and define a Tuple with richer semantics.

 Bye,
 bearophile


That's a shame.  I feel like this (along with functions and delgates 
being different) is one of the areas where the complexity of the 
language really shows itself.  I really liked the proposals you 
presented though.
Jul 01 2010
prev sibling next sibling parent Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Thu, Jul 1, 2010 at 01:13, bearophile <bearophileHUGS lycos.com> wrote:


 I'd like to add five enhancement requests in Bugzilla, related to
 std.typecons.Tuple. I show them here first for possible comments or
 critiques.


OK, here I go.

I don't plan to make you like current D tuples, but I'd like to show some
facilities the current syntax provide... and some things I discovered while
answering this, in case other people reading this didn't know them  either.


I have seen that std.typecons.Tuple is quite useful, but it misses some very

 important features. Some features can just be added to it (see for example
 bug 4381 )



Yes this one (adding a .length member) seems easy.

just add either

 property size_t length() { return Types.length; }

or

immutable length = Types.length





 and maybe the apply(). But other features can't just be added, they need
 some syntax and semantic support.

 Some other useful things:
 - More integration and usage of Tuple in druntime. For example the
 associative array property AA.byItem() can yield tuple(key,value) lazily,
 and AA.items can return a dynamic array of them.


I like that, as the range associated to AA is quite naturally a lazy
Tuple!(K,V)[].
opSlice() is not defined for AA, so whe not use it to return a range?

auto range = aa[]; // lazily produce (K,V) pairs




 - Some way to reverse the order of the items of a tuple (generating a tuple
 of different type).


I had some fun with this and ideas like this. Inserting elements, rotating
tuples, reversing them, etc. Even mapping polymorphic functions on tuples
and reducing them, though I don't think these should be part of any standard
library.

You can these there:
http://svn.dsource.org/projects/dranges/trunk/dranges/docs/tuple2.html


- "in" operator for tuples (as for arrays).


Yes, and that seems easily implementable. I use this:

bool contains(U, T...)(Tuple!T tup, U elem)
{
    static if (staticIndexOf!(U,T) == -1)
        return false;
    else
    {
        foreach(i, Type; tup.Types)
        {
            static if (is(Type == U))
                if (tup.field[i] == elem) return true;
        }
        return false;
    }
}

But I now see I could iterate directly by jumping at staticIndexOf(U,T) and
if the value is not OK, continuing on the tail. Though on most cases, tuples
have only a few elements...






 - More tuple-aware functions in Phobos, like a function to build an AA from
 a range of tuple(key, value).


Oh yes.



 - Optional: Zip and other pairing ranges to yield tuple (currently for
 efficiency they yield a pair of pointers. But this breaks abstraction. Using
 a more common data structure has significant advantages. The compiler can
 recognize the idiom and in some cases can avoid the creation of the Tuples
 that Zip produce.


I don't like the idea of compiler magic, but I do like the idea of having a
Zip that uses the standard structure: tuples. Its easier to inferface with
other functions this way. The Proxy from phobos zip is good for sorting
(which is why Andrei defined it this way, I guess), but it's a pain to use
otherwise, because it's a one-of-a-kind struct that cannot be fed to other
functions.




 Tuple unpacking is quite handy (the following syntax is just an
 syntax-idea, other syntaxes can be used. This syntax is not valid in C, so I
 think this syntax can be used in D):

 auto foo() {
    return tuple(10, "hello");
 }
 void main() {
    (int n, string s) = foo();


I also woud like that. While waiting to convince Walter, we can use some
poor man alternative like this one:

void copy(T...)(ref T to, Tuple!T from)
{
    foreach(i,Type; T)
    {
        to[i] = from.field[i];
    }
}

usage:

auto t = tuple(1, 3.14, "abc");
int i; double d; string s;
copy(i,d,s, t);

otherwise, you can use this:

auto ids = t.expand;

ids is an instantiated typetuple: you can index it (ids[1]), slice it
(ids[0..2]), iterate on it with foreach, etc.
The only thing you cannot do is returning it from a function ...

I know it's not as good as what you want (where you define and assign you n
and s in one go).




    writeln(n, " ", s);
    (n, s) = foo(); // calls is again
    int[] arr = [1, 2, 3];
    (int a, int b, int c) = arr; // other kind of unpacking


I experimented with a struct called RefTuple, that took constructor values
by ref and that did the kind of destructuring you present here. I created it
with a function called _  (yes, just _).
This gave the following syntax:

int n; string s;
_(n,s) = foo();


------------

 This is the syntax that can be currently used:

 auto foo() {
    T1 alpha = computeIt1(...);
    T1 alpha = computeIt2(...);
    return Tuple!(T1, "alpha", T2, "beta")(alpha, beta);
    //return tuple(alpha, beta); // alternative
 }
 void main() {
    auto alpha_beta = foo();
    use1(alpha_beta.alpha);
    use2(alpha_beta.beta);
    use1(alpha_beta.field[0]); // alternative
    use2(alpha_beta.field[1]);  // alternative
 }


You can also use ._x:

use1(alpha_beta._0);
use1(alpha_beta._1);




 ------------

 More syntax that can be currently used:

 import std.stdio, std.typecons;
 void main() {
    int[] arr2 = [tuple(1, 2, 3).tupleof];
    writeln(arr2);
 }

 But it prints:
 1 2 3 1 2 3
 Instead of:
 [1, 2, 3]


Yeah, tupleof return a strange value for tuple. I understood why, but
quickly forgot about it. Maybe it's possible to make .tupleof an alias of
.expand?
(** test, hmm no, doesn't work **)
I personnaly use expand a lot:

auto t = tuple(1,2,3);
int[] arr2 = [t.expand]; // works.

I like that to couple tuples and functions:

void foo(int i, double d, string s) {}

auto t = tuple(1, 3.14, "abc");

foo(t.expand); // works.



 opIndex/opIndexAssign syntax support for tuples:

 auto tup = tuple(10, 20, 30, 40);
 writeln(tup[0]);
 tup[1] = 5;
 int y = tup[2];
 tup[3]++;


 Slice syntax for tuples (currently done with 'Tuple.slice'):
 auto tup = tuple(10, 20, 30, 40, 50, 60);
 auto part = tup[1 .. 3];


Once again, you can use ._x, .field or .expand for that. It's not perfect,
but it's not bad either:

auto t = tuple(1, 3.14, "abc");
writeln(tup._0); // writes 1
t._1 = 1.414;       // t is now tuple(1, 1.414, "abc")
string s = t._2;
t._0++; // works


alternative syntax (.field or .expand)

writeln(t.field[0]);
t.field[1] = 1.414;
string s = t.field[2];
t.field[0]++;

And for slicing:

auto t2 = t.field[1..3]; //but t is an instantiated typetuple (ie, an 'old
way' tuple, a (int,string), not a std.typecons.Tuple!(int,string))
auto t3 = tuple(t.field[1..3]); t3 is a bone fide Tuple!(double, string)

Man, typetuples are almost perfect... If only they could once again be
returned from functions...





 The type of tuple elements can differ, so the index must be known at
 compile-time, just as with tupleof[].

 A possible way to implement it (currently this can't be used):
 alias this.tupleof[0..$] this;



Tuple code list bug 2800 as a blocker for this.




 ==========================================

 Title: [Better tuples] (static) foreach on tuple items

 This is one of the tuple enhancement requests. See bug xxxx for an
 introduction.

 auto tup = tuple(10, 20, 30);
 static foreach (item; tup)
    writeln(item);


So, you can use:

foreach(index, item; tup.expand)
    writeln(item);


(apply => smelting a tuple to give it to a function)

 A possible usage in D:

 void foo(T1, T2)(T1 x, T2 y) {
    writeln(x, " ", y);
 }
 void main() {
    auto tuple1 = tuple(1, 2);
    apply(&foo!(tuple1[0].typeof, tuple1[1].typeof), tuple1);
    auto tuple2 = tuple(1, 2, 3);
    apply(&foo!(tuple2[0].typeof, tuple2[1].typeof), tuple2[0..2]);
 }


Wheww.
OK, at the risk of repeating myself, why not like this?

void main() {
    auto t = tuple(1,"abc");
    foo(t.expand);
    auto t2 = tuple(3.14, 1, "abc");
    foo(t2.field[1..$]);
}

It's not that bad, no?


Philippe



 Jul 01 2010



prev sibling  parent  bearophile <bearophileHUGS lycos.com>  writes:


Thank you for your comments Philippe Sigaud, and sorry for my slow answer, I
was quite busy.



 just add either
  property size_t length() { return Types.length; }
 or
 immutable length = Types.length


Better:
enum length = Types.length;



 opSlice() is not defined for AA, so whe not use it to return a range?
 auto range = aa[]; // lazily produce (K,V) pairs


This is cute but I prefer something like AA.byItem() because it's more explicit
and readable.



Inserting elements, rotating tuples, reversing them, etc. Even mapping
polymorphic functions on tuples and reducing them, though I don't think these
should be part of any standard library.<


I think that slicing and concatenation are operations common enough to deserve
to be built-in in the stdandard tuples (slicing is already present, and my old
Record type has concatenation too).

Reverse of tuples is less commonly useful, and if tuples become well iterable
then you just need a generic reversed:
tuple(reverse(sometuple))
So a reverse specific for tuples is not useful...



- "in" operator for tuples (as for arrays).





Yes, and that seems easily implementable. I use this:<


Yep, but Walter doesn't like this semantics :-)



 - Optional: Zip and other pairing ranges to yield tuple (currently for
 efficiency they yield a pair of pointers. But this breaks abstraction. Using
 a more common data structure has significant advantages. The compiler can
 recognize the idiom and in some cases can avoid the creation of the Tuples
 that Zip produce.





I don't like the idea of compiler magic,


The idea of making the compiler more aware and able to optimize the zip range
better is positive, it's not magic, it's a compiler better able to digest the
language idioms, it's a natural thing. We have done this a bit in the ShedSkin
compiler because in Python code zip is used often.



 but I do like the idea of having a
 Zip that uses the standard structure: tuples. Its easier to inferface with
 other functions this way. The Proxy from phobos zip is good for sorting
 (which is why Andrei defined it this way, I guess), but it's a pain to use
 otherwise, because it's a one-of-a-kind struct that cannot be fed to other
 functions.


I agree :-)



 Once again, you can use ._x, .field or .expand for that.


The _x and .expand are not explained in this page, I didn't know about them:
http://www.digitalmars.com/d/2.0/phobos/std_typecons.html



 Tuple code list bug 2800 as a blocker for this.


Thank you. It's a bug reported by Andrei so probably he has tried the same idea
:-)



 foreach(index, item; tup.expand)
     writeln(item);


I prefer tuples to work more like arrays, so no expand. Less syntax to remember
and code to write.



 Wheww.
 OK, at the risk of repeating myself, why not like this?
 void main() {
     auto t = tuple(1,"abc");
     foo(t.expand);
     auto t2 = tuple(3.14, 1, "abc");
     foo(t2.field[1..$]);
 }


OK, the apply() is not so necessary.

Thank you for all your answers, you have improved some of my ideas :-)

Bye,
bearophile



 Jul 04 2010