www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Operator overloading or alternatives to expression templates

reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
I find the reasons for turining down my ER a bit moot.

[Issue 14593 – operator overloading can't be used with expression
templates](https://issues.dlang.org/show_bug.cgi?id=14593)

AFAIK expression templates are the primary choice tom implement SIMD and
matrix libraries.
And I still have [this idea](http://dpaste.dzfl.pl/cd375ac594cf) of
implementing a nice query language for ORMs.

D currently doesn't allow to override some operators like < <= > >= &&
|| !=.

At least the comparison operators are really limiting, e.g. it's not
possible to efficiently implement logical indexing.

vec[vec < 15], vec[vec == 15], vec[(vec != 15) & (vec > 10)]

Also opCmp is less efficient to implement than opBinary!"<" for many
types. Generally any good implementation of an algorithm should only
require a less operator, not a full ordering opCmp.

The short circuit operators && and || have a special semantic and can't
be easily, but there is & and | so it's not really required.

Now expression templates make an awful DSL when being used to create a
HTML formatter, but when the original semantic of the operators is
preserved they are really powerful b/c the compiler already handles
typechecking, operator precedence, and captures variables.

Does anyone have a different idea how to make a nice query language?
db.get!Person.where!(p => p.age > 21 && p.name == "Peter")
Sep 11 2015
next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, 11 September 2015 at 19:41:41 UTC, Martin Nowak wrote:
 Does anyone have a different idea how to make a nice query 
 language? db.get!Person.where!(p => p.age > 21 && p.name == 
 "Peter")
Don't the suggestions for DSLs generally revolve around using string mixins to use whatever syntax you want? - Jonathan M Davis
Sep 11 2015
prev sibling next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 09/11/2015 09:40 PM, Martin Nowak wrote:
 I find the reasons for turining down my ER a bit moot.
 ...
+1.
 [Issue 14593 – operator overloading can't be used with expression
 templates](https://issues.dlang.org/show_bug.cgi?id=14593)

 AFAIK expression templates are the primary choice tom implement SIMD and
 matrix libraries.
OTOH, they are a hack.
 And I still have [this idea](http://dpaste.dzfl.pl/cd375ac594cf) of
 implementing a nice query language for ORMs.

 D currently doesn't allow to override some operators like < <= > >= &&
 || !=.

 At least the comparison operators are really limiting, e.g. it's not
 possible to efficiently implement logical indexing.

 vec[vec < 15], vec[vec == 15], vec[(vec != 15) & (vec > 10)]

 Also opCmp is less efficient to implement than opBinary!"<" for many
 types. Generally any good implementation of an algorithm should only
 require a less operator, not a full ordering opCmp.
 ...
The rewrite a < b => a.opCmp(b)<0 is usually wasteful, but both methods can be the most efficient choice depending on the application. Two calls to opBinary!"<" will usually not be more efficient than one call to "opCmp". I.e. the precedence should be the other way: try a.opBinary!"<"(b) first and then fall back to a.opCmp(b)<0. (Maybe it would also make sense to automatically provide opCmp if missing when opBinary!"<" has been implemented, such that generic code can use opCmp for all comparable types. Built-in comparable types should also provide opCmp.)
 The short circuit operators && and || have a special semantic and can't
 be easily, but there is & and | so it's not really required.
 ...
There's the 'lazy' keyword.
 ...

 Does anyone have a different idea how to make a nice query language?
 db.get!Person.where!(p => p.age > 21 && p.name == "Peter")
You could give up on operator syntax.
Sep 11 2015
parent "Martin Nowak" <code dawg.eu> writes:
On Friday, 11 September 2015 at 23:19:54 UTC, Timon Gehr wrote:
 Does anyone have a different idea how to make a nice query 
 language?
 db.get!Person.where!(p => p.age > 21 && p.name == "Peter")
You could give up on operator syntax.
That's what I did in the prototype, p.age.gt(21), but it's somewhat ugly.
Sep 12 2015
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/11/2015 03:40 PM, Martin Nowak wrote:
 I find the reasons for turining down my ER a bit moot.

 [Issue 14593 – operator overloading can't be used with expression
 templates](https://issues.dlang.org/show_bug.cgi?id=14593)

 AFAIK expression templates are the primary choice tom implement SIMD and
 matrix libraries.
 And I still have [this idea](http://dpaste.dzfl.pl/cd375ac594cf) of
 implementing a nice query language for ORMs.
Expression templates are interesting, but from experience with them in C++ they're more trouble than they're worth. They haven't made much inroads in C++ outside exotic libraries because (a) they have odd and random limitations and corner cases and (b) they have really byzantine failure modes. Just look at any non-toy-example C++ use of ETs - it's completely bizarre. I'd say if a language wants to support expression templates properly, there's a lot of careful design to get into it, way beyond intercepting operators. Like e.g. C#/LINQ. I don't think D is prepared for that.
 D currently doesn't allow to override some operators like < <= > >= &&
 || !=.

 At least the comparison operators are really limiting, e.g. it's not
 possible to efficiently implement logical indexing.

 vec[vec < 15], vec[vec == 15], vec[(vec != 15) & (vec > 10)]

 Also opCmp is less efficient to implement than opBinary!"<" for many
 types. Generally any good implementation of an algorithm should only
 require a less operator, not a full ordering opCmp.

 The short circuit operators && and || have a special semantic and can't
 be easily, but there is & and | so it's not really required.

 Now expression templates make an awful DSL when being used to create a
 HTML formatter, but when the original semantic of the operators is
 preserved they are really powerful b/c the compiler already handles
 typechecking, operator precedence, and captures variables.

 Does anyone have a different idea how to make a nice query language?
 db.get!Person.where!(p => p.age > 21 && p.name == "Peter")
There's two canonical ways to do that. 1. Use lambdas, which seem to already do what you want: db.get!Person.filter!(p => p.age > 21 && p.name == "Peter") The way this'd go, the db.get!Person() call returns an input range of Person. Presumably introspection is being used to bind fields in the query such as "age" and "name" to static field names in struct Person. Then good old std.algorithm.filter takes care of the rest. 2. If you want to embed real SQL into D, use string-based DSLs. Andrei
Sep 11 2015
next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Fri, Sep 11, 2015 at 07:47:42PM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:
 On 09/11/2015 03:40 PM, Martin Nowak wrote:
I find the reasons for turining down my ER a bit moot.

[Issue 14593 – operator overloading can't be used with expression
templates](https://issues.dlang.org/show_bug.cgi?id=14593)

AFAIK expression templates are the primary choice tom implement SIMD
and matrix libraries.
And I still have [this idea](http://dpaste.dzfl.pl/cd375ac594cf) of
implementing a nice query language for ORMs.
Expression templates are interesting, but from experience with them in C++ they're more trouble than they're worth. They haven't made much inroads in C++ outside exotic libraries because (a) they have odd and random limitations and corner cases and (b) they have really byzantine failure modes. Just look at any non-toy-example C++ use of ETs - it's completely bizarre.
A number of years ago I wrote a computational application using Blitz++, an ET-based matrix / multidimensional array C++ library, and I have to say that the experience was mostly pleasant. However, it did push the limits of operator overloading abuse, such as: Array<2,int> matrix = Array<2,int>(4, 4); matrix = 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1; It also had unexpected reference/copying semantics (operator=() copies by reference, but `=` in a variable declarationi copies by value), as well as the occasional wrinkle when assigning the result of an expression template to a variable (sometimes you have to explicitly call a function to turn it into something storable in a variable). The error messages however, were completely inscrutable, just as you said. They generally begin anywhere from 15-16 lines of compiler output per error, and only grows from there. Needless to say, I did not have the patience (nor persistence!) to decipher those error messages; most of my efforts lay in copying textbook examples from the documentation and modifying them piece by piece, checking their compilability at every step, until they matched what I ultimately wanted. Writing anything complex directly was an invitation to be faced with an incomprehensible screen-filling error message (often more than one), and endless hours of randomly modifying random bits of syntax in hopes that the error will somehow, magically, go away.
 I'd say if a language wants to support expression templates properly,
 there's a lot of careful design to get into it, way beyond
 intercepting operators. Like e.g. C#/LINQ. I don't think D is prepared
 for that.
Pluggable syntax modules seem like an attractive idea, though. But probably outside the scope of D2 at present. [...]
Does anyone have a different idea how to make a nice query language?
db.get!Person.where!(p => p.age > 21 && p.name == "Peter")
There's two canonical ways to do that. 1. Use lambdas, which seem to already do what you want: db.get!Person.filter!(p => p.age > 21 && p.name == "Peter") The way this'd go, the db.get!Person() call returns an input range of Person. Presumably introspection is being used to bind fields in the query such as "age" and "name" to static field names in struct Person. Then good old std.algorithm.filter takes care of the rest. 2. If you want to embed real SQL into D, use string-based DSLs.
[...] Yeah, string-based DSLs make more sense in D: avoid complicating the core language, thorny issues surrounding operator overloading (and the abuse thereof), and also allow fully-free syntax of your choice. With CTFE, you can parse just about any DSL with any syntax at compile-time, thus incur none of the performance issues that runtime string-based DSLs would have. T -- The only difference between male factor and malefactor is just a little emptiness inside.
Sep 11 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/11/2015 08:03 PM, H. S. Teoh via Digitalmars-d wrote:
 Needless to say, I did not have
 the patience (nor persistence!) to decipher those error messages; most
 of my efforts lay in copying textbook examples from the documentation
 and modifying them piece by piece, checking their compilability at every
 step, until they matched what I ultimately wanted. Writing anything
 complex directly was an invitation to be faced with an incomprehensible
 screen-filling error message (often more than one), and endless hours of
 randomly modifying random bits of syntax in hopes that the error will
 somehow, magically, go away.
All the more ironic considering Blitz++ was intended for scientists who were not supposed to be programmer experts. -- Andrei
Sep 11 2015
parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 12 September 2015 at 00:27:45 UTC, Andrei 
Alexandrescu wrote:
 On 09/11/2015 08:03 PM, H. S. Teoh via Digitalmars-d wrote:
 Needless to say, I did not have
 the patience (nor persistence!) to decipher those error 
 messages; most
 of my efforts lay in copying textbook examples from the 
 documentation
 and modifying them piece by piece, checking their 
 compilability at every
 step, until they matched what I ultimately wanted. Writing 
 anything
 complex directly was an invitation to be faced with an 
 incomprehensible
 screen-filling error message (often more than one), and 
 endless hours of
 randomly modifying random bits of syntax in hopes that the 
 error will
 somehow, magically, go away.
All the more ironic considering Blitz++ was intended for scientists who were not supposed to be programmer experts. -- Andrei
I do think it says more about C++ than it does about expression templates.
Sep 11 2015
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-09-12 01:47, Andrei Alexandrescu wrote:

 There's two canonical ways to do that.

 1. Use lambdas, which seem to already do what you want:

 db.get!Person.filter!(p => p.age > 21 && p.name == "Peter")

 The way this'd go, the db.get!Person() call returns an input range of
 Person. Presumably introspection is being used to bind fields in the
 query such as "age" and "name" to static field names in struct Person.
 Then good old std.algorithm.filter takes care of the rest.
A database is really good at what it does. Therefore one wants the filter/where statement to be executed by the database, not the application code. It would also be quite wasteful to have to create an instance of Person to make the comparison then realizing it doesn't fulfill the condition and throw it away. I'm not sure how database protocols work but one definitely don't want to query the database for each row or return all the rows from the database.
 2. If you want to embed real SQL into D, use string-based DSLs.
The point is to _not_ use raw SQL. -- /Jacob Carlborg
Sep 12 2015
prev sibling parent reply "Martin Nowak" <code dawg.eu> writes:
On Friday, 11 September 2015 at 23:47:42 UTC, Andrei Alexandrescu 
wrote:
 1. Use lambdas, which seem to already do what you want:

 db.get!Person.filter!(p => p.age > 21 && p.name == "Peter")

 The way this'd go, the db.get!Person() call returns an input 
 range of Person. Presumably introspection is being used to bind 
 fields in the query such as "age" and "name" to static field 
 names in struct Person. Then good old std.algorithm.filter 
 takes care of the rest.
I'm instantiating the lambda with a fake p to capture the expression so I can translate it to whatever SQL, MongoDB, columnar db is used.
 2. If you want to embed real SQL into D, use string-based DSLs.
Strings don't capture context, aren't typechecked, and require a complex CTFE parser. db.get!Person.where!"age > 21 & name = ?"(name);
Sep 12 2015
next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 12-Sep-2015 23:08, Martin Nowak wrote:
 On Friday, 11 September 2015 at 23:47:42 UTC, Andrei Alexandrescu wrote:
 1. Use lambdas, which seem to already do what you want:

 db.get!Person.filter!(p => p.age > 21 && p.name == "Peter")

 The way this'd go, the db.get!Person() call returns an input range of
 Person. Presumably introspection is being used to bind fields in the
 query such as "age" and "name" to static field names in struct Person.
 Then good old std.algorithm.filter takes care of the rest.
I'm instantiating the lambda with a fake p to capture the expression so I can translate it to whatever SQL, MongoDB, columnar db is used.
 2. If you want to embed real SQL into D, use string-based DSLs.
Strings don't capture context, aren't typechecked, and require a complex CTFE parser. db.get!Person.where!"age > 21 & name = ?"(name);
What if we add generic string interpolation a-la: s"$age > 21 && name = ${someobj.field}" would translate to: AliasSeq!("",age," > 21 && name = ", someobj.field, ""); Thoughts? I think it was proposed before by Timothy Cour. -- Dmitry Olshansky
Sep 12 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/12/2015 04:08 PM, Martin Nowak wrote:
 On Friday, 11 September 2015 at 23:47:42 UTC, Andrei Alexandrescu wrote:
 1. Use lambdas, which seem to already do what you want:

 db.get!Person.filter!(p => p.age > 21 && p.name == "Peter")

 The way this'd go, the db.get!Person() call returns an input range of
 Person. Presumably introspection is being used to bind fields in the
 query such as "age" and "name" to static field names in struct Person.
 Then good old std.algorithm.filter takes care of the rest.
I'm instantiating the lambda with a fake p to capture the expression so I can translate it to whatever SQL, MongoDB, columnar db is used.
Yah, understood. Problem here is the approach is bound to run into walls at every few steps. Say you fix the comparisons to work. Then you have operators such as LIKE that are necessary yet not representable in D. So more tricks are in order. This is what I've seen with ET in C++ - an endless collection of tricks to achieve modest outcomes at enormous costs.
 2. If you want to embed real SQL into D, use string-based DSLs.
Strings don't capture context, aren't typechecked, and require a complex CTFE parser. db.get!Person.where!"age > 21 & name = ?"(name);
Understood as well. I figure from the example you have binding in mind, which is good. At some point you need to bite the bullet and write a parser for the relational query language you want there. Andrei
Sep 12 2015
next sibling parent deadalnix <deadalnix gmail.com> writes:
On Sunday, 13 September 2015 at 03:03:44 UTC, Andrei Alexandrescu 
wrote:
 On 09/12/2015 04:08 PM, Martin Nowak wrote:
 On Friday, 11 September 2015 at 23:47:42 UTC, Andrei 
 Alexandrescu wrote:
 1. Use lambdas, which seem to already do what you want:

 db.get!Person.filter!(p => p.age > 21 && p.name == "Peter")

 The way this'd go, the db.get!Person() call returns an input 
 range of
 Person. Presumably introspection is being used to bind fields 
 in the
 query such as "age" and "name" to static field names in 
 struct Person.
 Then good old std.algorithm.filter takes care of the rest.
I'm instantiating the lambda with a fake p to capture the expression so I can translate it to whatever SQL, MongoDB, columnar db is used.
Yah, understood. Problem here is the approach is bound to run into walls at every few steps. Say you fix the comparisons to work. Then you have operators such as LIKE that are necessary yet not representable in D. So more tricks are in order. This is what I've seen with ET in C++ - an endless collection of tricks to achieve modest outcomes at enormous costs.
Once gain, that says more about C++ than anything else. C# have such mechnism and users seems very happy with it.
 2. If you want to embed real SQL into D, use string-based 
 DSLs.
Strings don't capture context, aren't typechecked, and require a complex CTFE parser. db.get!Person.where!"age > 21 & name = ?"(name);
Understood as well. I figure from the example you have binding in mind, which is good. At some point you need to bite the bullet and write a parser for the relational query language you want there.
That sounds like a bigger problem than whatever you mentioned before.
Sep 12 2015
prev sibling next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Sunday, 13 September 2015 at 03:03:44 UTC, Andrei Alexandrescu 
wrote:
 Yah, understood. Problem here is the approach is bound to run 
 into walls at every few steps. Say you fix the comparisons to 
 work. Then you have operators such as LIKE that are necessary
like(Person.Name,"peter%") Person.Name == pattern("peter%")
 yet not representable in D. So more tricks are in order. This 
 is what I've seen with ET in C++ - an endless collection of 
 tricks to achieve modest outcomes at enormous costs.
FWIW, Guido wrote a similar query language to what Martin proposes for Google App Engine called NDB: https://cloud.google.com/appengine/docs/python/ndb/queries They already had a string query language called GQL: https://cloud.google.com/appengine/docs/python/datastore/gqlreference I never use GQL, because the NDB solution is superior in every way. GQL is only for the REPL.
 2. If you want to embed real SQL into D, use string-based 
 DSLs.
Strings don't capture context, aren't typechecked, and require a complex CTFE parser. db.get!Person.where!"age > 21 & name = ?"(name);
Understood as well. I figure from the example you have binding in mind, which is good. At some point you need to bite the bullet and write a parser for the relational query language you want there.
No. D should stop abusing strings. Strings should never contain references to type names. It makes it impossible to transpile D in the general case. Walter is unlikely to change his mind on opCmp, because he has put a lot of emphasis on that restriction since D1, but I think he overreacted on some C++ libraries being less than user friendly. The only thing you achieve is that people will create even uglier hacks to get what they want, like using a thread local global AST root pointer and build the AST as hidden side effects at runtime instead.
Sep 12 2015
prev sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 09/13/2015 05:03 AM, Andrei Alexandrescu wrote:
 Yah, understood. Problem here is the approach is bound to run into walls
 at every few steps. Say you fix the comparisons to work. Then you have
 operators such as LIKE that are necessary yet not representable in D. So
 more tricks are in order. This is what I've seen with ET in C++ - an
 endless collection of tricks to achieve modest outcomes at enormous costs.
But this is not an argument to rule out `opBinary!"<"`. To summarize the arguments. - logical indexing x[x < 20] e.g. opBinary!"<" returns a bit mask to select entries of a large vector - faster comparison struct Foo { size_t id; int opCmp(Foo rhs) { if (id < rhs.id) return -1; if (id == rhs.id) return 0; else return 1; } bool opBinary(string s:"<")(Foo rhs) { return id < rhs.id; } } Sorting a million Foos w/ random ids is 37.5% slower with opCmp. foos.sort!((a, b) => a.opBinary!"<"(b))(); // 104ms foos.sort!((a, b) => a < b)(); // 143ms - expression templates I'm well aware of the limitations, but still think it will work out nicely for an ORM b/c there is precedence in other language, e.g. Rails' (ActiveRecord) query syntax. - language regularization It's surprising to find these "arbitrary" language limitations. The non-predictability of what's possible has always been a huge issue for me with C++, i.e. you come up with an idea, spend 4 hours implementing it only to find out that the small detail x isn't feasible.
Sep 13 2015
next sibling parent Jack Stouffer <jack jackstouffer.com> writes:
On Sunday, 13 September 2015 at 14:06:46 UTC, Martin Nowak wrote:
 - logical indexing x[x < 20]

   e.g. opBinary!"<" returns a bit mask to select entries of a 
 large vector
Oh yes please! This is one of the more powerful features of numpy.
Sep 13 2015
prev sibling next sibling parent reply Daniel N <ufo orbiting.us> writes:
On Sunday, 13 September 2015 at 14:06:46 UTC, Martin Nowak wrote:
   struct Foo
   {
       size_t id;
       int opCmp(Foo rhs)
       {
           if (id < rhs.id) return -1;
           if (id == rhs.id) return 0;
           else return 1;
       }
       bool opBinary(string s:"<")(Foo rhs)
       {
           return id < rhs.id;
       }
   }

   Sorting a million Foos w/ random ids is 37.5% slower with 
 opCmp.
Could you try this? int opCmp(Foo rhs) { return (id > rhs.id) - (id < rhs.id); }
Sep 13 2015
next sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 09/13/2015 07:16 PM, Daniel N wrote:
 
 Could you try this?
 
 int opCmp(Foo rhs)
 {
   return (id > rhs.id) - (id < rhs.id);
 }
That's not the point, opCmp requires twice as many comparisons as needed for <. If they are more expansive, e.g. string comparison, your trick won't work.
Sep 13 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/13/2015 03:06 PM, Martin Nowak wrote:
 On 09/13/2015 07:16 PM, Daniel N wrote:
 Could you try this?

 int opCmp(Foo rhs)
 {
    return (id > rhs.id) - (id < rhs.id);
 }
That's not the point, opCmp requires twice as many comparisons as needed for <. If they are more expansive, e.g. string comparison, your trick won't work.
Well for string comparisons, the added cost (if not optimized away) is constant. Overall it seems to me that D is to be appreciated for requiring only one operator for all ordering comparisons (it causes a lot of noise in C++). Reducing comparisons is the job of the optimizer. Expression templates are a possible loss, but of a questionable trick. -- Andrei
Sep 14 2015
prev sibling next sibling parent reply burjui <bytefu gmail.com> writes:
On Sunday, 13 September 2015 at 17:16:40 UTC, Daniel N wrote:
 int opCmp(Foo rhs)
 {
   return (id > rhs.id) - (id < rhs.id);
 }
IMO, subtracting boolean values is bad code style, it's better to be explicit about your intention:
 (id > rhs.id ? 1 : 0) - (id < rhs.id ? 1 : 0)
Sep 14 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/14/2015 07:32 AM, burjui wrote:
 On Sunday, 13 September 2015 at 17:16:40 UTC, Daniel N wrote:
 int opCmp(Foo rhs)
 {
   return (id > rhs.id) - (id < rhs.id);
 }
IMO, subtracting boolean values is bad code style
I love it! -- Andrei
Sep 14 2015
parent deadalnix <deadalnix gmail.com> writes:
On Monday, 14 September 2015 at 18:24:00 UTC, Andrei Alexandrescu 
wrote:
 On 09/14/2015 07:32 AM, burjui wrote:
 On Sunday, 13 September 2015 at 17:16:40 UTC, Daniel N wrote:
 int opCmp(Foo rhs)
 {
   return (id > rhs.id) - (id < rhs.id);
 }
IMO, subtracting boolean values is bad code style
I love it! -- Andrei
Beside loving, it, this has good ILP, no branches, and have nice codegen on most CPUs. I was surprised to find that style in jemalloc, and compared to various alternatives. The only way I was able to get better is by ignoring (and by extension getting invalid results) overflow.
Sep 14 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/13/2015 01:16 PM, Daniel N wrote:
 On Sunday, 13 September 2015 at 14:06:46 UTC, Martin Nowak wrote:
   struct Foo
   {
       size_t id;
       int opCmp(Foo rhs)
       {
           if (id < rhs.id) return -1;
           if (id == rhs.id) return 0;
           else return 1;
       }
       bool opBinary(string s:"<")(Foo rhs)
       {
           return id < rhs.id;
       }
   }

   Sorting a million Foos w/ random ids is 37.5% slower with opCmp.
Could you try this? int opCmp(Foo rhs) { return (id > rhs.id) - (id < rhs.id); }
Apparently that does well with gcc: http://stackoverflow.com/questions/10996418/efficient-integer-compare-function -- Andrei
Sep 14 2015
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 09/14/2015 08:18 PM, Andrei Alexandrescu wrote:
 On 09/13/2015 01:16 PM, Daniel N wrote:
 On Sunday, 13 September 2015 at 14:06:46 UTC, Martin Nowak wrote:
   struct Foo
   {
       size_t id;
       int opCmp(Foo rhs)
       {
           if (id < rhs.id) return -1;
           if (id == rhs.id) return 0;
           else return 1;
       }
       bool opBinary(string s:"<")(Foo rhs)
       {
           return id < rhs.id;
       }
   }

   Sorting a million Foos w/ random ids is 37.5% slower with opCmp.
Could you try this? int opCmp(Foo rhs) { return (id > rhs.id) - (id < rhs.id); }
Apparently that does well with gcc: http://stackoverflow.com/questions/10996418/efficient-integer-compare-function -- Andrei
The first version was actually slightly faster when I tested it with std::sort. (i.e. if only "<" is used). I guess it is optimized to a simple integer comparison after inlining. In any case, any performance differences for such simple equivalent functions are random quirks of the respective back-ends.
Sep 14 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/13/2015 10:06 AM, Martin Nowak wrote:
 On 09/13/2015 05:03 AM, Andrei Alexandrescu wrote:
 Yah, understood. Problem here is the approach is bound to run into walls
 at every few steps. Say you fix the comparisons to work. Then you have
 operators such as LIKE that are necessary yet not representable in D. So
 more tricks are in order. This is what I've seen with ET in C++ - an
 endless collection of tricks to achieve modest outcomes at enormous costs.
But this is not an argument to rule out `opBinary!"<"`.
Agreed.
 To summarize the arguments.

 - logical indexing x[x < 20]

    e.g. opBinary!"<" returns a bit mask to select entries of a large vector
Good one. But niche, can be solved with a function, etc.
 - faster comparison

    struct Foo
    {
        size_t id;
        int opCmp(Foo rhs)
        {
            if (id < rhs.id) return -1;
            if (id == rhs.id) return 0;
            else return 1;
        }
        bool opBinary(string s:"<")(Foo rhs)
        {
            return id < rhs.id;
        }
    }

    Sorting a million Foos w/ random ids is 37.5% slower with opCmp.

    foos.sort!((a, b) => a.opBinary!"<"(b))(); // 104ms
    foos.sort!((a, b) => a < b)(); // 143ms
I think this is a good candidate for a peephole optimization.
 - expression templates

    I'm well aware of the limitations, but still think it will work out
 nicely for an ORM b/c there is precedence in other language, e.g. Rails'
 (ActiveRecord) query syntax.
Fine, but let's not forget the ET experience in D will be a lot closer to that in C++ than that in Rails. And that doesn't bode well.
 - language regularization

    It's surprising to find these "arbitrary" language limitations.
    The non-predictability of what's possible has always been a huge issue
 for me with C++, i.e. you come up with an idea, spend 4 hours
 implementing it only to find out that the small detail x isn't feasible.
This is the case in all powerful languages. Martin Odersky told me there's a constant problem with Scala having such a powerful type systems, the envelope of what can and cannot be done with it is really fuzzy and frustrates its power users. Andrei
Sep 14 2015
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 09/14/2015 08:09 PM, Andrei Alexandrescu wrote:
 On 09/13/2015 10:06 AM, Martin Nowak wrote:
 ...
 - language regularization

    It's surprising to find these "arbitrary" language limitations.
    The non-predictability of what's possible has always been a huge issue
 for me with C++, i.e. you come up with an idea, spend 4 hours
 implementing it only to find out that the small detail x isn't feasible.
This is the case in all powerful languages.
That's an overgeneralization. Furthermore, having arbitrary designed-in irregularities is not comparable to implementing a system whose emergent behavior is not understood sufficiently well. (D is guilty of both, but the former is much easier to fix than the latter.)
 Martin Odersky told me there's a constant problem with Scala ...
Both C++ and Scala have accidentally Turing-complete type systems.
Sep 14 2015
next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Monday, 14 September 2015 at 19:35:39 UTC, Timon Gehr wrote:
 Furthermore, having arbitrary designed-in irregularities is not 
 comparable to implementing a system whose emergent behavior is 
 not understood sufficiently well. (D is guilty of both, but the 
 former is much easier to fix than the latter.)
Well, D1 was a restrictive design that made C++ style programming simpler and less error prone (sans compiler bugs). D2 is a flexible and complicated package and thereby those restrictions cause disharmony. It is basically too incompatible design philosophies that are brought together. Like in D1 it would make sense to say "the language provide the essentials, don't design your own pointer types", whereas in D2 it makes sense to say "all features should be library based". I'm a bit miffed that it is difficult to built proper smart pointer types in D2.
 Both C++ and Scala have accidentally Turing-complete type 
 systems.
Yes, not sure how smart that is in terms of maintainable code. There is a reaction against overdone type systems that sometimes can be more of burden as programs age/grow (e.g. C++/Boost), so now we see gradual typing, deliberately unsound type-systems etc.
Sep 14 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/14/2015 03:35 PM, Timon Gehr wrote:
 On 09/14/2015 08:09 PM, Andrei Alexandrescu wrote:
 On 09/13/2015 10:06 AM, Martin Nowak wrote:
 ...
 - language regularization

    It's surprising to find these "arbitrary" language limitations.
    The non-predictability of what's possible has always been a huge
 issue
 for me with C++, i.e. you come up with an idea, spend 4 hours
 implementing it only to find out that the small detail x isn't feasible.
This is the case in all powerful languages.
That's an overgeneralization.
Aren't they all :o).
 Furthermore, having arbitrary designed-in irregularities is not
 comparable to implementing a system whose emergent behavior is not
 understood sufficiently well. (D is guilty of both, but the former is
 much easier to fix than the latter.)

 Martin Odersky told me there's a constant problem with Scala ...
Both C++ and Scala have accidentally Turing-complete type systems.
I think what I'm trying to say is "I'm trying to do this very advanced thing and the language support is insufficient" is the kind of argument that needs to be made and taken with caution. Andrei
Sep 15 2015
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 09/15/2015 06:53 PM, Andrei Alexandrescu wrote:
 On 09/14/2015 03:35 PM, Timon Gehr wrote:
 On 09/14/2015 08:09 PM, Andrei Alexandrescu wrote:
 On 09/13/2015 10:06 AM, Martin Nowak wrote:
 ...
 - language regularization

    It's surprising to find these "arbitrary" language limitations.
    The non-predictability of what's possible has always been a huge
 issue
 for me with C++, i.e. you come up with an idea, spend 4 hours
 implementing it only to find out that the small detail x isn't
 feasible.
This is the case in all powerful languages.
That's an overgeneralization.
Aren't they all :o). ...
I guess we have now established that there are at least two of them. :P
 Furthermore, having arbitrary designed-in irregularities is not
 comparable to implementing a system whose emergent behavior is not
 understood sufficiently well. (D is guilty of both, but the former is
 much easier to fix than the latter.)

 Martin Odersky told me there's a constant problem with Scala ...
Both C++ and Scala have accidentally Turing-complete type systems.
I think what I'm trying to say is "I'm trying to do this very advanced thing and the language support is insufficient" is the kind of argument that needs to be made and taken with caution. ...
This is certainly the case, and I believe the necessary care has been applied. The language support required isn't actually very advanced, nor are all usage scenarios. Arbitrary restrictions are usually design mistakes.
Sep 17 2015
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-09-11 21:40, Martin Nowak wrote:
 I find the reasons for turining down my ER a bit moot.

 [Issue 14593 – operator overloading can't be used with expression
 templates](https://issues.dlang.org/show_bug.cgi?id=14593)

 AFAIK expression templates are the primary choice tom implement SIMD and
 matrix libraries.
 And I still have [this idea](http://dpaste.dzfl.pl/cd375ac594cf) of
 implementing a nice query language for ORMs.
I would really like this.
 The short circuit operators && and || have a special semantic and can't
 be easily, but there is & and | so it's not really required.
Not sure if this would be a problem in D but there's a library in Ruby that does exactly this. Due to the operator precedence you end up needing to wrap everything in parentheses: Person.where{ |p| (p.age > 21) & (p.name == "Peter") | (p.name == "Foo") }
 Does anyone have a different idea how to make a nice query language?
 db.get!Person.where!(p => p.age > 21 && p.name == "Peter")
AST macros ;) -- /Jacob Carlborg
Sep 12 2015
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 11 September 2015 at 19:41:41 UTC, Martin Nowak wrote:
 Does anyone have a different idea how to make a nice query 
 language? db.get!Person.where!(p => p.age > 21 && p.name == 
 "Peter")
Do it at runtime?
Sep 12 2015
prev sibling next sibling parent reply Bahman Movaqar <Bahman BahmanM.com> writes:
On Fri, 11 Sep 2015 21:40:58 +0200, Martin Nowak 
<code+news.digitalmars dawg.eu> wrote:
 Does anyone have a different idea how to make a nice query language?
 db.get!Person.where!(p => p.age > 21 && p.name == "Peter")
Django's approach is, IMO, the cleverest and least magical one while keeping it expressive and efficient: Person.objects.filter(age__gt=21, name__eq='peter') The input of `filter` is keyword argument style. So, filter can access the dynamic argument names, in addition to their values. This makes it easy to understand user's intent without the need to overloaded operators or black magic (like repeated reflections Hibernate uses) and the syntax stays clean from user's perspective as well. -- -- Bahman Movaqar
Sep 12 2015
parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Saturday, 12 September 2015 at 22:38:53 UTC, Bahman Movaqar 
wrote:
 Django's approach is, IMO,  the cleverest and least magical one 
 while keeping it expressive and efficient:

     Person.objects.filter(age__gt=21, name__eq='peter')
As the main examples in this thread are for ORMs, I think the best case for more powerful operator overloading is the very popular Python library SQLAlchmey. SQLAlchmey is the reason that any project of mine that requires a database is going to be in Python. Here is a real code sample from one of my projects from my day job: shipped = session.query( func.sum( InvoiceHistoryDetail.QuantityShipped * InvoiceHistoryDetail.UnitPrice ).label("sum") ).join( InvoiceHistoryHeader, InvoiceHistoryHeader.InvoiceNo == InvoiceHistoryDetail.InvoiceNo ).join( Customer, and_( InvoiceHistoryHeader.ARDivisionNo == Customer.ARDivisionNo, InvoiceHistoryHeader.CustomerNo == Customer.CustomerNo ) ).filter( InvoiceHistoryHeader.ShipDate >= fiscal_month.PeriodStartingDate, InvoiceHistoryHeader.ShipDate == datetime.date.today() - datetime.timedelta(days=1), InvoiceHistoryHeader.SalesOrderNo != None, Customer.UDF_PARTICIPATION.in_(participation) ).one() Note how complex operator overloading is used to make very readable JOIN and WHERE clauses. One other cool thing to note is the func function, which is a generator function that allows expressing any SQL function in Python by translating the name after the dot to the function name is the final SQL. So func.whatever(model.field) would become WHATEVER(`model.field`) in the SQL. I know that this effect is much harder to create in a explicitly and strongly typed language, but I just wanted to show a real world example of how these could be used to great effect.
Sep 13 2015
parent Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 09/13/2015 03:15 PM, Jack Stouffer wrote:
 
 I know that this effect is much harder to create in a explicitly and
 strongly typed language, but I just wanted to show a real world example
 of how these could be used to great effect.
But it is doable in D, and even better it's possible to optimize the queries b/c we can know in advance what fields are used. http://dpaste.dzfl.pl/cd375ac594cf
Sep 13 2015
prev sibling next sibling parent reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig rejectedsoftware.com> writes:
Am 11.09.2015 um 21:40 schrieb Martin Nowak:
 I find the reasons for turining down my ER a bit moot.

 [Issue 14593 – operator overloading can't be used with expression
 templates](https://issues.dlang.org/show_bug.cgi?id=14593)

 (...)
The missing support for overloading the individual relational operators is something that has often turned out very limiting to me in the past, too. The opCmp approach is good as a default, but explicitly disallowing implementation of the individual operators seems arbitrary.
 Now expression templates make an awful DSL when being used to create a
 HTML formatter, but when the original semantic of the operators is
 preserved they are really powerful b/c the compiler already handles
 typechecking, operator precedence, and captures variables.

 Does anyone have a different idea how to make a nice query language?
 db.get!Person.where!(p => p.age > 21 && p.name == "Peter")
I had played around with some ideas for a similar project, but didn't find a really satisfying solution: https://github.com/rejectedsoftware/dotter/blob/11ec72325e76c3329a58545526940c1df5328a2d/source/dotter/orm.d#L320
Sep 13 2015
parent Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 09/13/2015 11:00 AM, Sönke Ludwig wrote:
 I had played around with some ideas for a similar project, but didn't
 find a really satisfying solution:
 https://github.com/rejectedsoftware/dotter/blob/11ec72325e76c3329a58545526940c1df5328a2d/source/dotter/orm.d#L320
Yeah, that doesn't look too nice. I think db.get!Author.where!(a => a.books.count > 10) is very powerful b/c you can easily use relations for querying. struct Author { hasMany // relation in SQL or nested in MongoDB Book[] books; }
Sep 13 2015
prev sibling next sibling parent Dicebot <public dicebot.lv> writes:
I am pretty sure I'd prefer hygienic string DSL (with explicit 
alias list for used context symbols, like in Diet) over LINQ-like 
magic.
Sep 13 2015
prev sibling next sibling parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Friday, 11 September 2015 at 19:41:41 UTC, Martin Nowak wrote:
 Does anyone have a different idea how to make a nice query 
 language? db.get!Person.where!(p => p.age > 21 && p.name == 
 "Peter")
In our last project we took the following approach: `auto q = query.builder!Person.age!">"(20).name("Peter");` In the query-builders there was a lot of ugly concatenating sql stuff. But there was just one guy writing that, everybody else was using the high-level functions. It allows for a lot of checking, not only on types but also on logic (e.g. age!">"(20).age!"<"(10) was caught at runtime). Plus, very unittestable. It worked so well that once the query-builders were written, all the constraints, sorting and ordering parameters could be passed straight from the http api down to the query-builders. Still, you would have to write the query-builders for each Object you want query, although that is something you could do in a DSL.
Sep 14 2015
next sibling parent Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 09/14/2015 03:47 PM, Sebastiaan Koppe wrote:
 
 In our last project we took the following approach:
 
 `auto q = query.builder!Person.age!">"(20).name("Peter");`
Interesting idea.
Sep 14 2015
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 14 September 2015 at 13:47:10 UTC, Sebastiaan Koppe 
wrote:
 `auto q = query.builder!Person.age!">"(20).name("Peter");`
I confess that I'm not really paying attention to this thread, but I can't help but think plain old literal: `Person.age > 20 && Person.name = 'Peter'` is nicer. You can still CT check that by parsing the string if you want.
Sep 14 2015
parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Monday, 14 September 2015 at 18:17:05 UTC, Adam D. Ruppe wrote:
 On Monday, 14 September 2015 at 13:47:10 UTC, Sebastiaan Koppe 
 wrote:
 `auto q = query.builder!Person.age!">"(20).name("Peter");`
I confess that I'm not really paying attention to this thread, but I can't help but think plain old literal: `Person.age > 20 && Person.name = 'Peter'` is nicer. You can still CT check that by parsing the string if you want.
It is definitely nicer, but this is also a contrived use-case. What happens when you have a couple of joins, how would you write that? Or inner queries? Suppose we wanted to fetch all users who didn't place an order in the last year, it would be as simple as calling `.lastOrderDate!"<"(lastYear)`. It would do the join with the order table on the appropriate column. Granted, someone has to write `lastOrderDate()`.
Sep 14 2015
prev sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Friday, 11 September 2015 at 19:41:41 UTC, Martin Nowak wrote:
 AFAIK expression templates are the primary choice tom implement 
 SIMD and
 matrix libraries.
 And I still have [this 
 idea](http://dpaste.dzfl.pl/cd375ac594cf) of
 implementing a nice query language for ORMs.
While expression templates are used in many matrix libraries, there has also been some move away from them. Blaze uses so-called smart expression templates because of what they view as performance limitations of expression templates. I think the general idea is to reduce the creation of temporaries and ordering the calculations so as to minimize the size of the problem (like in the multiplication A*B*c, doing A*(B*c) instead of (A*B)*c) http://arxiv.org/pdf/1104.1729.pdf
 At least the comparison operators are really limiting, e.g. 
 it's not possible to efficiently implement logical indexing.

 vec[vec < 15], vec[vec == 15], vec[(vec != 15) & (vec > 10)]
I loves the logical indexing.
Sep 14 2015