www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Proposal: Database Engine for D

reply Piotrek <nodata nodata.pl> writes:
The goal of this post is to measure the craziness of an idea to 
embed a database engine into the D language ;)

I think about a database engine which would meet my three main 
requirements:
   - integrated with D (ranges)
   - ACID
   - fast

Since the days when I was working on financing data SW I become 
allergic to SQL. I though that NoSQL databases would fill the 
bill. Unfortunately they didn't. And I want to have an ability to 
write a code like this without too much effort:

   struct Person
   {
    string name;
    string surname;
    ubyte age;
    Address address;
   }

  DataBase db = new DataBase("file.db");
  auto coll = db.collection!Person("NSA.Registry");
  auto visitationList = coll.filter!(p => p.name == "James");
  writeln (visitationList);

And other things like updating and deleting from db. I think you 
get my point.

So I started a PoC project based on SQLite design:
https://github.com/PiotrekDlang/AirLock/blob/master/docs/database/design.md#architecture

The PoC code: 
https://github.com/PiotrekDlang/AirLock/tree/master/src/database

Can you please share your thoughts and experience on the topic? 
Has anyone tried similar things?

Piotrek
Dec 31 2015
next sibling parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 01/01/16 6:14 AM, Piotrek wrote:
 The goal of this post is to measure the craziness of an idea to embed a
 database engine into the D language ;)

 I think about a database engine which would meet my three main
 requirements:
    - integrated with D (ranges)
    - ACID
    - fast

 Since the days when I was working on financing data SW I become allergic
 to SQL. I though that NoSQL databases would fill the bill. Unfortunately
 they didn't. And I want to have an ability to write a code like this
 without too much effort:

    struct Person
    {
     string name;
     string surname;
     ubyte age;
     Address address;
    }

   DataBase db = new DataBase("file.db");
   auto coll = db.collection!Person("NSA.Registry");
   auto visitationList = coll.filter!(p => p.name == "James");
   writeln (visitationList);

 And other things like updating and deleting from db. I think you get my
 point.

 So I started a PoC project based on SQLite design:
 https://github.com/PiotrekDlang/AirLock/blob/master/docs/database/design.md#architecture


 The PoC code:
 https://github.com/PiotrekDlang/AirLock/tree/master/src/database

 Can you please share your thoughts and experience on the topic? Has
 anyone tried similar things?

 Piotrek
You've just introduced two topics. The first is a database engine, abstracting away the drivers. And second an ORM.
Dec 31
parent Piotrek <nodata nodata.pl> writes:
On Friday, 1 January 2016 at 01:34:53 UTC, Rikki Cattermole wrote:
 You've just introduced two topics.
 The first is a database engine, abstracting away the drivers.
 And second an ORM.
And maybe even an object-oriented database management system to some extent. OTOH, I removed SQL from the stack. Piotrek
Jan 02
prev sibling next sibling parent reply tcak <1ltkrs+3wyh1ow7kzn1k sharklasers.com> writes:
On Thursday, 31 December 2015 at 17:14:55 UTC, Piotrek wrote:
 The goal of this post is to measure the craziness of an idea to 
 embed a database engine into the D language ;)

 I think about a database engine which would meet my three main 
 requirements:
   - integrated with D (ranges)
   - ACID
   - fast

 Since the days when I was working on financing data SW I become 
 allergic to SQL. I though that NoSQL databases would fill the 
 bill. Unfortunately they didn't. And I want to have an ability 
 to write a code like this without too much effort:

   struct Person
   {
    string name;
    string surname;
    ubyte age;
    Address address;
   }

  DataBase db = new DataBase("file.db");
  auto coll = db.collection!Person("NSA.Registry");
  auto visitationList = coll.filter!(p => p.name == "James");
  writeln (visitationList);

 And other things like updating and deleting from db. I think 
 you get my point.

 So I started a PoC project based on SQLite design:
 https://github.com/PiotrekDlang/AirLock/blob/master/docs/database/design.md#architecture

 The PoC code: 
 https://github.com/PiotrekDlang/AirLock/tree/master/src/database

 Can you please share your thoughts and experience on the topic? 
 Has anyone tried similar things?

 Piotrek
You know someone needs to maintain all that code base continuously. When SQLite is a separate project, it has its own developers and we just bind to its library; it is same for other DBs. Your proposal is nice, but creating another standard, and a group of people needs to take care of it in both documentation and coding wise continuously are biggest issues.
Dec 31
parent Piotrek <nodata nodata.pl> writes:
On Friday, 1 January 2016 at 04:20:19 UTC, tcak wrote:
 You know someone needs to maintain all that code base 
 continuously. When SQLite is a separate project, it has its own 
 developers and we just bind to its library; it is same for 
 other DBs. Your proposal is nice, but creating another 
 standard, and a group of people needs to take care of it in 
 both documentation and coding wise continuously are biggest 
 issues.
Agree. Lack of resources is an enemy of every project. BTW. I don't introduce any new standard except a new file format. Piotrek
Jan 02
prev sibling next sibling parent reply Kapps <opantm2+spam gmail.com> writes:
On Thursday, 31 December 2015 at 17:14:55 UTC, Piotrek wrote:
   struct Person
   {
    string name;
    string surname;
    ubyte age;
    Address address;
   }

  DataBase db = new DataBase("file.db");
  auto coll = db.collection!Person("NSA.Registry");
  auto visitationList = coll.filter!(p => p.name == "James");
  writeln (visitationList);
This example shows the difficulty of doing this in D. You can't really have something like `p.Name == "James"`, or `p.Age < 21` translate to SQL properly without language changes, which I believe Walter or Andrei were against. This has been the key problem when things like Linq to Sql for D have been brought up before.
Jan 01
next sibling parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Fri, 2016-01-01 at 10:00 +0000, Kapps via Digitalmars-d wrote:
 On Thursday, 31 December 2015 at 17:14:55 UTC, Piotrek wrote:
=20
 =C2=A0 struct Person
 =C2=A0 {
 =C2=A0=C2=A0=C2=A0string name;
 =C2=A0=C2=A0=C2=A0string surname;
 =C2=A0=C2=A0=C2=A0ubyte age;
 =C2=A0=C2=A0=C2=A0Address address;
 =C2=A0 }
=20
 =C2=A0DataBase db =3D new DataBase("file.db");
 =C2=A0auto coll =3D db.collection!Person("NSA.Registry");
 =C2=A0auto visitationList =3D coll.filter!(p =3D> p.name =3D=3D "James"=
);
 =C2=A0writeln (visitationList);
=20 This example shows the difficulty of doing this in D. You can't=C2=A0 really have something like `p.Name =3D=3D "James"`, or `p.Age < 21`=C2=A0 translate to SQL properly without language changes, which I=C2=A0 believe Walter or Andrei were against. This has been the key=C2=A0 problem when things like Linq to Sql for D have been brought up=C2=A0 before.
Why does it need language changes? Having the ability to have an internal DSL instead of SQL string fiddling is one of the major wins for SQLAlchemy. If it can be done in Python why can't it be done in D? --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Jan 01
next sibling parent reply Kapps <opantm2+spam gmail.com> writes:
On Friday, 1 January 2016 at 10:26:14 UTC, Russel Winder wrote:
 On Fri, 2016-01-01 at 10:00 +0000, Kapps via Digitalmars-d 
 wrote:
 On Thursday, 31 December 2015 at 17:14:55 UTC, Piotrek wrote:
 
   struct Person
   {
    string name;
    string surname;
    ubyte age;
    Address address;
   }
 
  DataBase db = new DataBase("file.db");
  auto coll = db.collection!Person("NSA.Registry");
  auto visitationList = coll.filter!(p => p.name == "James");
  writeln (visitationList);
This example shows the difficulty of doing this in D. You can't really have something like `p.Name == "James"`, or `p.Age < 21` translate to SQL properly without language changes, which I believe Walter or Andrei were against. This has been the key problem when things like Linq to Sql for D have been brought up before.
Why does it need language changes? Having the ability to have an internal DSL instead of SQL string fiddling is one of the major wins for SQLAlchemy. If it can be done in Python why can't it be done in D?
Someone else can explain better / more correctly than me, but I believe the issue lies with opCmp and opEquals. You can make expressions like p.Name.equals("James") work (I believe using opDispatch), but because all you have is opEquals, you can't know if the user put in 'p.Name == "James"` or `p.Name != "James"`, as they both simply call opEquals. In order to do that, you would need things like opLessThan, opEquals, opNotEquals, opGreaterThan, etc, which would (with improper use or bugs) cause other issues, like a < b && a > b and a == b && a != b to be true, or a == b || a != b to be false. I'm also not certain how you could implement `p => p.Name == "James" || p.Name == "Bob"`, but there might be a way? I think this is the gist of it, but I'm likely wrong on some aspects of this, so it would be good if someone else clarified..
Jan 01
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On Friday, 1 January 2016 at 10:40:59 UTC, Kapps wrote:

 Someone else can explain better / more correctly than me, but I 
 believe the issue lies with opCmp and opEquals. You can make 
 expressions like p.Name.equals("James") work (I believe using 
 opDispatch), but because all you have is opEquals, you can't 
 know if the user put in 'p.Name == "James"` or `p.Name != 
 "James"`, as they both simply call opEquals. In order to do 
 that, you would need things like opLessThan, opEquals, 
 opNotEquals, opGreaterThan, etc, which would (with improper use 
 or bugs) cause other issues, like a < b && a > b and a == b && 
 a != b to be true, or a == b || a != b to be false.

 I'm also not certain how you could implement `p => p.Name == 
 "James" || p.Name == "Bob"`, but there might be a way? I think 
 this is the gist of it, but I'm likely wrong on some aspects of 
 this, so it would be good if someone else clarified..
That's exactly the problem. See an issue reported [1] and a related thread [2]. AST macros could solve the problem as well [3] (and many other problems). [1] https://issues.dlang.org/show_bug.cgi?id=14593 [2] http://forum.dlang.org/thread/msvapl$2rmn$1 digitalmars.com [3] http://wiki.dlang.org/DIP50 -- /Jacob Carlborg
Jan 01
parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Fri, 2016-01-01 at 11:03 +0000, Jacob Carlborg via Digitalmars-d
wrote:
=20
[=E2=80=A6]
 That's exactly the problem. See an issue reported [1] and a=C2=A0
 related thread [2]. AST macros could solve the problem as well=C2=A0
 [3] (and many other problems).
=20
 [1] https://issues.dlang.org/show_bug.cgi?id=3D14593
 [2] http://forum.dlang.org/thread/msvapl$2rmn$1 digitalmars.com
 [3] http://wiki.dlang.org/DIP50
Or alternative 4, fix D so that proper operator definition can be achieved. =C2=A0 --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Jan 01
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2016-01-01 12:33, Russel Winder via Digitalmars-d wrote:
 On Fri, 2016-01-01 at 11:03 +0000, Jacob Carlborg via Digitalmars-d
 wrote:

 […]
 That's exactly the problem. See an issue reported [1] and a
 related thread [2]. AST macros could solve the problem as well
 [3] (and many other problems).

 [1] https://issues.dlang.org/show_bug.cgi?id=14593
 [2] http://forum.dlang.org/thread/msvapl$2rmn$1 digitalmars.com
 [3] http://wiki.dlang.org/DIP50
Or alternative 4, fix D so that proper operator definition can be achieved.
That would be the first link and the second link discussing that issue. -- /Jacob Carlborg
Jan 02
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/1/2016 3:33 AM, Russel Winder via Digitalmars-d wrote:
 Or alternative 4, fix D so that proper operator definition can be
 achieved.
The way D's operator overloading is designed is not a mistake. It's limitations are a feature, not a bug. It was deliberately set up to: 1. Make doing C++ style iostreams hard. 2. Prevent clever use of operator overloading and expression templates to create languages that look like D, but are NOT. 3. Work well when using operator overloading to implement arithmetic types. For example, I've seen operator overloading used in C++ to turn it into a sort-of regex language. The failures of it are: 1. Sort-of because C++ operator precedence and prefix/postfix grammar is different than that of regex, so it can't be emulated correctly. 2. It is visually indistinguishable from C++ code. You simply cannot look at a piece of code and tell it is regex with utterly different meaning, rather than the usual meanings. 3. Any error messages from misuse are utterly and totally incomprehensible, because the compiler is designed to compile C++ and gives C++ messages, not regex messages. 4. C++ ETs are legendary in their tendency to consume all the memory in the computer and take incredibly long to compile. Most fail when the expressions exceed a rather small level of complexity because of this. My not-so-humble opinion is these sorts of DSLs are technical demonstrations, but not useful nor desirable tools. *************************************** So, what does D do? http://dlang.org/phobos/std_regex.html D enables CTFE to write a PROPER compile time regex language, with correct regex grammar, correct regex tokens, regex-specific error messages, etc. And it works great! It's not a hack, compromise, or workaround. It's a real, embedded DSL. And it's faster than any other regex engine on the market.
Jan 04
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Monday, 4 January 2016 at 09:44:35 UTC, Walter Bright wrote:
 My not-so-humble opinion is these sorts of DSLs are technical 
 demonstrations, but not useful nor desirable tools.
Well, this is wrong. They are desirable and work with tooling, editors, code completion etc. And also allow user extensions if done right. Symbols should not be referenced as strings. That's bad. That's textual macro processing.
Jan 04
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/4/2016 2:34 AM, Ola Fosheim Grøstad wrote:
 On Monday, 4 January 2016 at 09:44:35 UTC, Walter Bright wrote:
 My not-so-humble opinion is these sorts of DSLs are technical demonstrations,
 but not useful nor desirable tools.
Well, this is wrong. They are desirable and work with tooling, editors, code completion etc. And also allow user extensions if done right.
You didn't address my points. Have you used C++ ETs for production? They fall apart in real world use for the reasons I mentioned. It's one thing to read a paper about something, and another to use it for something larger than will fit on a slide.
Jan 04
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Monday, 4 January 2016 at 11:19:38 UTC, Walter Bright wrote:
ou didn't address my points. Have you used C++ ETs for
 production? They fall apart in real world use for the reasons I 
 mentioned. It's one thing to read a paper about something, and 
 another to use it for something larger than will fit on a slide.
You are assuming template expressions. I am not. The main overhead when interacting with databases is network traffic, building the query should be done at runtime. No need for templates. I am in favour of having a clean core language (minimal language after desugaring/lowering) that can be manipulated using term rewriting/symbolic manipulation (or ast macros). But that sounds more like D4 than D2...?
Jan 04
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2016-01-04 10:44, Walter Bright wrote:

 So, what does D do?

     http://dlang.org/phobos/std_regex.html

 D enables CTFE to write a PROPER compile time regex language, with
 correct regex grammar, correct regex tokens, regex-specific error
 messages, etc. And it works great! It's not a hack, compromise, or
 workaround. It's a real, embedded DSL. And it's faster than any other
 regex engine on the market.
And how would you do that for SQL. Build a CTFE SQL database? -- /Jacob Carlborg
Jan 04
prev sibling next sibling parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 2016-01-04 at 01:44 -0800, Walter Bright via Digitalmars-d
wrote:
 On 1/1/2016 3:33 AM, Russel Winder via Digitalmars-d wrote:
 Or alternative 4, fix D so that proper operator definition can be
 achieved.
=20 The way D's operator overloading is designed is not a mistake. It's limitations=C2=A0 are a feature, not a bug. It was deliberately set up to: =20 1. Make doing C++ style iostreams hard.
What is the problem with having the << and >> operators do input output. Very object-oriented.=C2=A0
 2. Prevent clever use of operator overloading and expression
 templates to create=C2=A0
 languages that look like D, but are NOT.
But it would be D. Boost.Sprint code may look like EBNF, but it is C++.
 3. Work well when using operator overloading to implement arithmetic
 types.
It is important that this works. But it should be possible to create an operator algebra for any type: arithmetic types are a very small subset of types used in computing.
 For example, I've seen operator overloading used in C++ to turn it
 into a=C2=A0
 sort-of regex language. The failures of it are:
=20
 1. Sort-of because C++ operator precedence and prefix/postfix grammar
 is=C2=A0
 different than that of regex, so it can't be emulated correctly.
=20
 2. It is visually indistinguishable from C++ code. You simply cannot
 look at a=C2=A0
 piece of code and tell it is regex with utterly different meaning,
 rather than=C2=A0
 the usual meanings.
=20
 3. Any error messages from misuse are utterly and totally
 incomprehensible,=C2=A0
 because the compiler is designed to compile C++ and gives C++
 messages, not=C2=A0
 regex messages.
=20
 4. C++ ETs are legendary in their tendency to consume all the memory
 in the=C2=A0
 computer and take incredibly long to compile. Most fail when the
 expressions=C2=A0
 exceed a rather small level of complexity because of this.
=20
 My not-so-humble opinion is these sorts of DSLs are technical
 demonstrations,=C2=A0
 but not useful nor desirable tools.
=20
 ***************************************
 So, what does D do?
=20
 =C2=A0=C2=A0=C2=A0=C2=A0http://dlang.org/phobos/std_regex.html
=20
 D enables CTFE to write a PROPER compile time regex language, with
 correct regex=C2=A0
 grammar, correct regex tokens, regex-specific error messages, etc.
 And it works=C2=A0
 great! It's not a hack, compromise, or workaround. It's a real,
 embedded DSL.=C2=A0
 And it's faster than any other regex engine on the market.
I avoid regular expressions except when editing using ed, sed, or emacs, so I cannot properly comment on the above. However I find Boost.Sprint a very sensible way of using compile-time meta-object protocols. You write a grammar in EBNF in the source code, and the compiler writes the parser. Excellent use of internal DSL. And type safe. --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Jan 04
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/4/2016 10:25 AM, Russel Winder via Digitalmars-d wrote:
 On Mon, 2016-01-04 at 01:44 -0800, Walter Bright via Digitalmars-d
 wrote:
 1. Make doing C++ style iostreams hard.
What is the problem with having the << and >> operators do input output. Very object-oriented.
I'm surprised you'd ask. I thought this was well known. 1. ugly as hell 2. not exception safe 3. not thread safe 4. interleaved output in the middle of lines when writing to both stdout and stderr 2..4 are caused by inability to encapsulate a sequence of these operations.
 2. Prevent clever use of operator overloading and expression
 templates to create
 languages that look like D, but are NOT.
But it would be D. Boost.Sprint code may look like EBNF, but it is C++.
#define BEGIN { #define END } is still C++, too.
 3. Work well when using operator overloading to implement arithmetic
 types.
It is important that this works. But it should be possible to create an operator algebra for any type: arithmetic types are a very small subset of types used in computing.
What do you suggest when the operators and precedence of the desired algebraic type simply do not map cleanly onto C++ operators and precedence grammar? Allow users to define their own operators and redefine the precedence? Where is the line that shouldn't be crossed?
 I avoid regular expressions except when editing using ed, sed, or
 emacs, so I cannot properly comment on the above. However I find
 Boost.Sprint a very sensible way of using compile-time meta-object
 protocols. You write a grammar in EBNF in the source code, and the
 compiler writes the parser. Excellent use of internal DSL. And type
 safe.
EBNF doesn't map onto C++ operators and precedence, either. It's still a hack: https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_Form With CFTE, the user can actually write actual EBNF.
Jan 04
next sibling parent Wyatt <wyatt.epp gmail.com> writes:
On Monday, 4 January 2016 at 20:07:55 UTC, Walter Bright wrote:
 On 1/4/2016 10:25 AM, Russel Winder via Digitalmars-d wrote:
 It is important that this works. But it should be possible to 
 create an
 operator algebra for any type: arithmetic types are a very 
 small subset
 of types used in computing.
What do you suggest when the operators and precedence of the desired algebraic type simply do not map cleanly onto C++ operators and precedence grammar? Allow users to define their own operators and redefine the precedence? Where is the line that shouldn't be crossed?
On this point specifically, I'm still considering attempting to write a DIP because it would be very useful to have some user-defined operators available. Take the OCaml/F# approach where they provide a set of acceptable characters for operators and define precedence automatically (depending on first character of the name). (And definitely don't do the Haskell thing where any binary function can be turned into an infix op.) -Wyatt
Jan 04
prev sibling parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 2016-01-04 at 12:07 -0800, Walter Bright via Digitalmars-d
wrote:
=20
[=E2=80=A6]
 I'm surprised you'd ask. I thought this was well known.
=20
 1. ugly as hell
For you perhaps, others have different opinions. I like it.
 2. not exception safe
Indeed. but then most code is not exception safe which is part of the problem with computing. Fixing C++ I/O is only about 0.001% of the problem.
 3. not thread safe
Indeed. But then a properly architected system only has one point of I/O per channel, which could then be single threaded, so less of a problem.
 4. interleaved output in the middle of lines when writing to both
 stdout and stderr
This is a shell problem not a C++ problem.
 2..4 are caused by inability to encapsulate a sequence of these
 operations.
And also generally bad architecting of I/O in applications. [=E2=80=A6]
=20
+.
=20
 =C2=A0 #define BEGIN {
 =C2=A0 #define END }
=20
 is still C++, too.
I thought Stephen Bourne was the only person who ever thought this was sensible. There is a difference always between what you can do and what you should do. The language should not be fascist in stopping all things not deemed the one true way by the language designers. To do so stops serendipitous evolution. [=E2=80=A6]
=20
 What do you suggest when the operators and precedence of the desired
 algebraic=C2=A0
 type simply do not map cleanly onto C++ operators and precedence
 grammar? Allow=C2=A0
 users to define their own operators and redefine the precedence?
 Where is the=C2=A0
 line that shouldn't be crossed?
There is no line. cf. Algol68.=C2=A0 Python only offers compiler understood operators and preserves precedence, ditto C++ (the relationship is fairly obvious). Scala allows arbitrary operators, cf. Algol68. Groovy follows the Python/C++ approach over Java. So currently most people stick with know operators and allow redefinition within a fixed precedence structure. Easy for the compiler writer, constraining for the user programmers, but workable usually. Scala has shown the way that Algol68 set out, compiler writers should give up on imposing fixed expression structures in their languages. Haskell has gone a little this way using `` for infix operator words. Sort of works can be ugly. Of course Scala code can be made incomprehensible using the operator definitions. In the end it is up to programmers to do the right thing, and for compiler writers not to tell them what they may or may not thing in a given language. =C2=A0Restrictive languages, lead to restrictive applications, and demise in the market. [=E2=80=A6]
=20
 EBNF doesn't map onto C++ operators and precedence, either. It's
 still a hack:
=20
 =C2=A0=C2=A0=C2=A0https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93N=
aur_Form
=20
 With CFTE, the user can actually write actual EBNF.
Assuming EBNF is actually a good way of writing grammers. ;-) --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Jan 09
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/9/2016 3:02 AM, Russel Winder via Digitalmars-d wrote:
 [...]
 And also generally bad architecting of I/O in applications.
These problems have persisted since the genesis of iostreams. The problems do not exist in the way other modern languages do I/O. Iostreams was a great idea in 1985, but 30 years of experience has shown there are better ways.
 There is a difference always between what you can do and what you
 should do. The language should not be fascist in stopping all things
 not deemed the one true way by the language designers. To do so stops
 serendipitous evolution.
On the other hand, consider the C preprocessor. It has well known problems. But this never seems to discourage the most execrable edifices built on top of it. Their designers inevitably regard them as shining examples of their creativity. And maybe they are. But do they belong in professional code? No. But the only way to stop them is to not have a preprocessor in the language. (Of course, you can always use m4 as a front end to any language, but thankfully nobody seems to do that.) I believe there is a notion of "best practices", that some practices are simply better than others. A good language design makes it easy to follow best practices, and hard to follow known inferior ones. D is meant for professional use, and people who want professional code tend to want some mechanical assurance that best practices are followed (consider the increasing smorgasbord of third party tools designed to ferret out bad practices in C/C++ code). There is plenty of innovation in programming language practice, I don't think it is stifled.
Jan 09
prev sibling next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Saturday, 9 January 2016 at 11:02:46 UTC, Russel Winder wrote:
 There is a difference always between what you can do and what 
 you should do. The language should not be fascist in stopping 
 all things not deemed the one true way by the language 
 designers. To do so stops serendipitous evolution.
Indeed, although I think languages should have a library-mode and an application-mode. That way you can constrain certain types of meta-programming to libraries and keep application implementation syntax simple and readable. This is what happens in well structured programs anyway.
Jan 09
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/9/16 6:02 AM, Russel Winder via Digitalmars-d wrote:
 Assuming EBNF is actually a good way of writing grammers.;-)
What would be a better way? -- Andrei
Jan 09
parent reply w0rp <devw0rp gmail.com> writes:
I've played with the idea of using operator overloading for some 
kind of ORM before, but I don't think it's strictly necessary to 
use operator overloading for an ORM at all. Maybe in some cases 
it might make sense.

I don't think the answer for building such a thing is to think of 
one idea, find out it won't work out well, and then give up. You 
have to be more creative than that.
Jan 12
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Tuesday, 12 January 2016 at 12:53:29 UTC, w0rp wrote:
 I've played with the idea of using operator overloading for 
 some kind of ORM before, but I don't think it's strictly 
 necessary to use operator overloading for an ORM at all. Maybe 
 in some cases it might make sense.
The question is whether we want to provide a comparable experience to other languages or not. What is desired depends on what other languages provide for a particular application area? The ability to create abstractions is important for a language like D since that is supposed to be a primary feature.
Jan 12
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 01/04/2016 01:25 PM, Russel Winder via Digitalmars-d wrote:
 What is the problem with having the << and >> operators do input
 output. Very object-oriented.
What is the connection between using shift operators for I/O and object orientation?
 But it would be D. Boost.Sprint code may look like EBNF, but it is C++.
[snip]
 I avoid regular expressions except when editing using ed, sed, or
 emacs, so I cannot properly comment on the above. However I find
 Boost.Sprint a very sensible way of using compile-time meta-object
 protocols. You write a grammar in EBNF in the source code, and the
 compiler writes the parser. Excellent use of internal DSL. And type
 safe.
I don't think regexen were the point as much as an example. What is Boost.Sprint? I haven't heard of it, and can't find it. Thanks! -- Andrei
Jan 04
next sibling parent Zz <zz nospam.com> writes:
 What is Boost.Sprint? I haven't heard of it, and can't find it. 
 Thanks! -- Andrei
I believe he meant Boost Spirit http://www.boost.org/doc/libs/1_60_0/libs/spirit/doc/html/spirit/introduction.html Zz
Jan 04
prev sibling next sibling parent Ettienne Gilbert <ettienne.gilbert gmail.com> writes:
On Monday, 4 January 2016 at 20:38:46 UTC, Andrei Alexandrescu 
wrote:
 On 01/04/2016 01:25 PM, Russel Winder via Digitalmars-d wrote:
[snip]
 What is Boost.Sprint? I haven't heard of it, and can't find it. 
 Thanks! -- Andrei
I think he miss-typed or miss-remembered. It is Boost.Spirit. http://boost-spirit.com/home/
Jan 05
prev sibling parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 2016-01-04 at 15:38 -0500, Andrei Alexandrescu via Digitalmars-
d wrote:
=20
[=E2=80=A6]
 I don't think regexen were the point as much as an example. What is=C2=A0
 Boost.Sprint? I haven't heard of it, and can't find it. Thanks!=C2=A0
As others mentioned, it was a mis-type, it is Boost.Spirit. http://boost-spirit.com/home/
=20
--=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Jan 07
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 01/07/2016 02:02 PM, Russel Winder via Digitalmars-d wrote:
 On Mon, 2016-01-04 at 15:38 -0500, Andrei Alexandrescu via Digitalmars-
 d wrote:

 […]
 I don't think regexen were the point as much as an example. What is
 Boost.Sprint? I haven't heard of it, and can't find it. Thanks!
As others mentioned, it was a mis-type, it is Boost.Spirit. http://boost-spirit.com/home/
Got it, thanks. I took a look at a familiar topic - json grammars. I just googled `boost spirit json` and followed through the top result. For C++, the grammar definition would be https://github.com/cierelabs/json_spirit/blob/master/ciere/json/parser/grammar_def.hpp. Then I googled `pegged json` and the top result led to https://github.com/PhilippeSigaud/Pegged/blob/master/pegged/examples/json.d. Which do you like more? Andrei
Jan 07
parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Thu, 2016-01-07 at 15:32 -0500, Andrei Alexandrescu via Digitalmars-
d wrote:
 On 01/07/2016 02:02 PM, Russel Winder via Digitalmars-d wrote:
[=E2=80=A6]
=20
 Got it, thanks. I took a look at a familiar topic - json grammars.
=20
 I just googled `boost spirit json` and followed through the top
 result.=C2=A0
 For C++, the grammar definition would be=C2=A0
 https://github.com/cierelabs/json_spirit/blob/master/ciere/json/parse
 r/grammar_def.hpp.
=20
 Then I googled `pegged json` and the top result led to=C2=A0
 https://github.com/PhilippeSigaud/Pegged/blob/master/pegged/examples/
 json.d.
=20
 Which do you like more?
Just a quick look I'm afraid, so very superficial at this stage: The D code as presented is clearly simpler and easy to read. However the D grammar is a string and not code so the anticipation is that errors in the C++ code would be easier to find and correct. I guess the question here is how good the compile time evaluation of strings is in D and does it do good error reporting. I still prefer the concept of Internal DSL explicitly in the code rather than in strings just now, but this may still just be prejudice. --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Jan 09
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/9/2016 2:49 AM, Russel Winder via Digitalmars-d wrote:
 The D code as presented is clearly simpler and easy to read. However
 the D grammar is a string and not code so the anticipation is that
 errors in the C++ code would be easier to find and correct. I guess the
 question here is how good the compile time evaluation of strings is in
 D and does it do good error reporting.
The compiler does not report the errors. The parsing is done by code supplied by the module, and the error messages are as good or as bad as that code is written. The point being that one is not subjected to compiler generated error messages based on the compiler thinking it is C++/D code, but a custom written parser for the DSL that can give messages specific to the DSL.
 I still prefer the concept of Internal DSL explicitly in the code
 rather than in strings just now, but this may still just be prejudice.
I can see only one advantage to using ETs for DSLs: * 3rd party tools like syntax directed editors, formatters and highlighters will operate on it. But they'll still operate on it as if it were C++ code. CTFE based DSLs win on every other point I can think of, including error messages.
Jan 09
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/9/16 5:49 AM, Russel Winder via Digitalmars-d wrote:
 On Thu, 2016-01-07 at 15:32 -0500, Andrei Alexandrescu via Digitalmars-
 d wrote:
 On 01/07/2016 02:02 PM, Russel Winder via Digitalmars-d wrote:
[…]
 Got it, thanks. I took a look at a familiar topic - json grammars.

 I just googled `boost spirit json` and followed through the top
 result.
 For C++, the grammar definition would be
 https://github.com/cierelabs/json_spirit/blob/master/ciere/json/parse
 r/grammar_def.hpp.

 Then I googled `pegged json` and the top result led to
 https://github.com/PhilippeSigaud/Pegged/blob/master/pegged/examples/
 json.d.

 Which do you like more?
Just a quick look I'm afraid, so very superficial at this stage: The D code as presented is clearly simpler and easy to read. However the D grammar is a string and not code so the anticipation is that errors in the C++ code would be easier to find and correct. I guess the question here is how good the compile time evaluation of strings is in D and does it do good error reporting.
The string parser is custom code and issues errors as defined by the library author. (I don't have experience with Pegged in particular.) I think one difficulty with ETs is reporting line where errors occur in multi-line strings. Language facilities could help here. This is an area of strategic importance for D. I do have extensive experience with error messages in C++ ETs, which were and are inscrutable.
 I still prefer the concept of Internal DSL explicitly in the code
 rather than in strings just now, but this may still just be prejudice.
Prejudice can last only so long in the face of evidence. Andrei
Jan 09
prev sibling parent rsw0x <anonymous anonymous.com> writes:
On Monday, 4 January 2016 at 09:44:35 UTC, Walter Bright wrote:
 On 1/1/2016 3:33 AM, Russel Winder via Digitalmars-d wrote:
 Or alternative 4, fix D so that proper operator definition can 
 be
 achieved.
The way D's operator overloading is designed is not a mistake. It's limitations are a feature, not a bug. It was deliberately set up to: 1. Make doing C++ style iostreams hard.
struct _cout { auto opBinary(string s : "<<", T)(auto ref T rhs) { import std.stdio : write; write(rhs); return this; } } _cout cout; void main() { cout << "Walter " << "hates " << "this." << endl; } and yet, I can't overload the address-of operator because Walter is afraid I might shoot my foot.
Jan 07
prev sibling parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Fri, 2016-01-01 at 10:40 +0000, Kapps via Digitalmars-d wrote:
 [=E2=80=A6]
=20
 Someone else can explain better / more correctly than me, but I=C2=A0
 believe the issue lies with opCmp and opEquals. You can make=C2=A0
 expressions like p.Name.equals("James") work (I believe using=C2=A0
 opDispatch), but because all you have is opEquals, you can't know=C2=A0
 if the user put in 'p.Name =3D=3D "James"` or `p.Name !=3D "James"`, as=
=C2=A0
 they both simply call opEquals. In order to do that, you would=C2=A0
 need things like opLessThan, opEquals, opNotEquals,=C2=A0
 opGreaterThan, etc, which would (with improper use or bugs) cause=C2=A0
 other issues, like a < b && a > b and a =3D=3D b && a !=3D b to be=C2=A0
 true, or a =3D=3D b || a !=3D b to be false.
=20
 I'm also not certain how you could implement `p =3D> p.Name =3D=3D=C2=A0
 "James" || p.Name =3D=3D "Bob"`, but there might be a way? I think=C2=A0
 this is the gist of it, but I'm likely wrong on some aspects of=C2=A0
 this, so it would be good if someone else clarified..
Hummm=E2=80=A6 so to put it another way, the D meta-object protocol is even more broken than that of Java: at least in Java there isn't even a pretence that you can create an internal DSL. This is very, very sad, I had not realized D was this broken. =C2=A0 --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Jan 01
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/1/16 6:30 AM, Russel Winder via Digitalmars-d wrote:
 On Fri, 2016-01-01 at 10:40 +0000, Kapps via Digitalmars-d wrote:
 […]

 Someone else can explain better / more correctly than me, but I
 believe the issue lies with opCmp and opEquals. You can make
 expressions like p.Name.equals("James") work (I believe using
 opDispatch), but because all you have is opEquals, you can't know
 if the user put in 'p.Name == "James"` or `p.Name != "James"`, as
 they both simply call opEquals. In order to do that, you would
 need things like opLessThan, opEquals, opNotEquals,
 opGreaterThan, etc, which would (with improper use or bugs) cause
 other issues, like a < b && a > b and a == b && a != b to be
 true, or a == b || a != b to be false.

 I'm also not certain how you could implement `p => p.Name ==
 "James" || p.Name == "Bob"`, but there might be a way? I think
 this is the gist of it, but I'm likely wrong on some aspects of
 this, so it would be good if someone else clarified..
Hummm… so to put it another way, the D meta-object protocol is even more broken than that of Java: at least in Java there isn't even a pretence that you can create an internal DSL. This is very, very sad, I had not realized D was this broken.
Well I guess it's time to cut losses and cancel that D tutorial at ACCU 2016, eh :o). I understand how the shortcomings of D's expression templates (ETs) for SQL (as compared to C++ or LINQ) can be construed as an indication of a language breakage, but I don't quite agree for a few reasons. One, D offers another mechanism aside from ETs for embedded domain-specific languages. So in a way there's less pressure on offering lush ET than would be in other languages. Second, ET as a mechanism for SQL interface has other inherent limitations. Consider the "LIKE" operator in SQL, which has no ET equivalent in C++ with similar syntax, and no direct equivalent in LINQ. That doesn't mean the respective languages are broken. Third, evaluating the merits and demerits of a language choice should be done within several appropriate use cases, not just one. For example, D's use of opEquals/opCmp saves a lot of boilerplate and potential bugs in many cases compared to C++. On another example, C++'s overloading of &&, ||, and the comma operator are considered downright disastrous and are unrecommended by virtually all coding standards and guidelines, yet do work with ETs. Andrei
Jan 03
next sibling parent reply rumbu <rumbu rumbu.ro> writes:
On Monday, 4 January 2016 at 00:49:29 UTC, Andrei Alexandrescu 
wrote:
 Second, ET as a mechanism for SQL interface has other inherent 
 limitations. Consider the "LIKE" operator in SQL, which has no 
 ET equivalent in C++ with similar syntax, and no direct 
 equivalent in LINQ. That doesn't mean the respective languages 
 are broken.
LINQ transforms StartsWith, EndsWith and Contains methods in LIKE queries under the hood since Entity Framework 4.0 (8 years ago). Also, when in doubt, there is an entire namespace dedicated to LINQ queries with direct SQL equivalents (including LIKE) - https://msdn.microsoft.com/en-us/library/system.data.linq.sqlclient.sqlmethods(v=vs.110).aspx
Jan 03
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 01/04/2016 01:49 AM, rumbu wrote:
 On Monday, 4 January 2016 at 00:49:29 UTC, Andrei Alexandrescu wrote:
 Second, ET as a mechanism for SQL interface has other inherent
 limitations. Consider the "LIKE" operator in SQL, which has no ET
 equivalent in C++ with similar syntax, and no direct equivalent in
 LINQ. That doesn't mean the respective languages are broken.
LINQ transforms StartsWith, EndsWith and Contains methods in LIKE queries under the hood since Entity Framework 4.0 (8 years ago). Also, when in doubt, there is an entire namespace dedicated to LINQ queries with direct SQL equivalents (including LIKE) - https://msdn.microsoft.com/en-us/library/system.data.linq.sqlclient.sqlmethods(v=vs.110).aspx
Misunderstanding. Please re-read what I wrote. -- Andrei
Jan 04
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2016-01-04 01:49, Andrei Alexandrescu wrote:

 Second, ET as a mechanism for SQL interface has other inherent
 limitations. Consider the "LIKE" operator in SQL, which has no ET
 equivalent in C++ with similar syntax, and no direct equivalent in LINQ.
 That doesn't mean the respective languages are broken.
It's one operator that doesn't have a direct corresponding in the host language. With D, it's not possible to handle most of the operators.
 Third, evaluating the merits and demerits of a language choice should be
 done within several appropriate use cases, not just one. For example,
 D's use of opEquals/opCmp saves a lot of boilerplate and potential bugs
 in many cases compared to C++.
If D allowed to separately overload the comparison operators the standard library should of course provide a mixin of some kind to reduce the boilerplate.
 On another example, C++'s overloading of
 &&, ||, and the comma operator are considered downright disastrous and
 are unrecommended by virtually all coding standards and guidelines, yet
 do work with ETs.
I think for generating SQL, overloading && and || would be a valid use case. There's of course AST macros as well, which have many other good use cases. Unfortunately you don't like those either :( -- /Jacob Carlborg
Jan 03
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/3/2016 11:40 PM, Jacob Carlborg wrote:
 There's of course AST macros as well, which have many other good use cases.
 Unfortunately you don't like those either :(
Neither Andrei nor I have changed our minds on that one.
Jan 04
parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 2016-01-04 at 01:45 -0800, Walter Bright via Digitalmars-d
wrote:
 On 1/3/2016 11:40 PM, Jacob Carlborg wrote:
 There's of course AST macros as well, which have many other good
 use cases.
 Unfortunately you don't like those either :(
=20 Neither Andrei nor I have changed our minds on that one.
Pity I like good AST transforms: Groovy, Scala, Lisp, Clojure, Rust=E2=80= =A6 C++ template meta-programming isn't really my cup of tea. Rusts macros show it can be done well. --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Jan 04
next sibling parent reply rsw0x <anonymous anonymous.com> writes:
On Monday, 4 January 2016 at 18:28:15 UTC, Russel Winder wrote:
 On Mon, 2016-01-04 at 01:45 -0800, Walter Bright via 
 Digitalmars-d wrote:
 On 1/3/2016 11:40 PM, Jacob Carlborg wrote:
 There's of course AST macros as well, which have many other 
 good
 use cases.
 Unfortunately you don't like those either :(
Neither Andrei nor I have changed our minds on that one.
Pity I like good AST transforms: Groovy, Scala, Lisp, Clojure, Rust… C++ template meta-programming isn't really my cup of tea. Rusts macros show it can be done well.
I agree heavily with this, it feels like an arbitrary limitation. D did wonders in making template metaprogramming usable compared to C++, but sometimes it feels like trying to pound nails in with a screwdriver. I think just looking at the kind of stuff being done in languages that enable AST macros might change minds.
Jan 04
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 01/04/2016 02:50 PM, rsw0x wrote:
 D did wonders in making template metaprogramming usable compared to C++,
 but sometimes it feels like trying to pound nails in with a screwdriver.
Do you have examples of that awkwardness happening? Is it suitable to supplement those with CTFE? -- Andrei
Jan 04
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 01/04/2016 01:28 PM, Russel Winder via Digitalmars-d wrote:
 Rusts macros show it can be done well.
Do you have a few examples handy? Thanks. -- Andrei
Jan 04
next sibling parent Charles Hixson via Digitalmars-d <digitalmars-d puremagic.com> writes:
FWIW, were I proposing a "Database Engine for D" I'd be proposing a 
B+Tree that was restricted to storing explicit data (no pointers or 
other indirection...including strings, you'd need to specify fixed size 
arrays of char, wchar, or dchar).  There would be one "type"/file, and 
the "type" would be given by a struct that contained no pointers either 
obvious or hidden (i.e., dynamic arrays, strings, etc.).  This would be 
for simplicity of implementation and basic functionality. Later I would 
consider implementing secondary keys which would be implemented as 
databases storing only the key value and the record number.  Since 
records are of fixed size, sorting them would be trivial, but only the 
secondary keys would provide this option because you want the record 
number of the main file to be reliable and invariant.

SQL seems like a poor fit to D.  At some point one might consider a 
fancier front end to the database that would allow indirections, but 
that's a whole can of worms, with threats of multiple copies that have 
different values for items that started out pointing to the same variable.

On 01/04/2016 12:39 PM, Andrei Alexandrescu via Digitalmars-d wrote:
 On 01/04/2016 01:28 PM, Russel Winder via Digitalmars-d wrote:
 Rusts macros show it can be done well.
Do you have a few examples handy? Thanks. -- Andrei
Jan 04
prev sibling parent Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 2016-01-04 at 15:39 -0500, Andrei Alexandrescu via Digitalmars-
d wrote:
 On 01/04/2016 01:28 PM, Russel Winder via Digitalmars-d wrote:
 Rusts macros show it can be done well.
=20 Do you have a few examples handy? Thanks. -- Andrei
Will do but it may be next week. Sorry, but medical stuff=E2=80=A6 --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Jan 07
prev sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Monday, 4 January 2016 at 00:49:29 UTC, Andrei Alexandrescu 
wrote:
 Second, ET as a mechanism for SQL interface has other inherent 
 limitations. Consider the "LIKE" operator in SQL, which has no 
 ET equivalent in C++ with similar syntax, and no direct
like(Person.Name,"peter%") Person.Name == pattern("peter%")
 On another example, C++'s overloading of &&, ||, and the comma 
 operator are considered downright disastrous and are 
 unrecommended by virtually all coding standards and guidelines,
Guideline means that you should think hard about it before you do break it. A guidelines does not mean that you never should do it. It means you should only break the guideline when you have good reasons to do so. If you can overload "and" and "or", you also can implement ternary logic, fuzzy logic and other logics. In some cases that is desirable. In C++ features were abused when they were new because people were eager to stretch new features to the limits. This is not a problem in regular C++ programs. And in the case of C++ iostreams, it has become an idiom that isn't causing any problems. C++ programmers are never confused by shift and output operators. But if this is a big deal the easy solution is to add unicode operators without default behaviour. String processing is not an acceptable alternative, it ruins tooling. You should essentially never be able to turn a symbol into a string and vice versa. D unfortunately allows you to do it. That is a mal-feature aka a textual macro feature. You can't really say that D has gotten rid of macros when you actually encourage using macro-like textual substitution features in place of processing proper symbols. That is an outdated 70s paradigm.
Jan 04
prev sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Friday, 1 January 2016 at 10:26:14 UTC, Russel Winder wrote:
 Why does it need language changes?

 Having the ability to have an internal DSL instead of SQL 
 string fiddling is one of the major wins for SQLAlchemy. If it 
 can be done in Python why can't it be done in D?
In D1 Walter made a point about restricting operator overloading to discourage reuse of operators. In D2 there are many ways to create your own weird syntax, but Walter is locked on his D1 position, even though it makes little sense in D2. This is an overarching theme in D: the design rationale, that goes a decade back and is based on making a restricted easy to use version of C++, does not change, even though the context is very different in 2015 and the premises has changed. These things are unlikely to change without a fork. Which is a pitty as D needs a more coherent design to get out of stagnation.
Jan 01
next sibling parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Fri, 2016-01-01 at 10:45 +0000, Ola Fosheim Gr=C3=B8stad via Digitalmars=
-
d wrote:
=20
[=E2=80=A6]
 In D1 Walter made a point about restricting operator overloading=C2=A0
 to discourage reuse of operators. In D2 there are many ways to=C2=A0
 [=E2=80=A6]
Java also went the route of "operator definition is too hard for programmers to deal with so we will not allow it". Every language on the JVM other than Java has made a point of allowing, indeed encouraging, sensible operator definition (*). =C2=A0Sadly too few people working on the JVM are allowed to use languages other than Java. (*) Arguably Scala takes this too far. --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Jan 01
next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Friday, 1 January 2016 at 11:37:40 UTC, Russel Winder wrote:
 Java also went the route of "operator definition is too hard 
 for programmers to deal with so we will not allow it". Every 
 language on the JVM other than Java has made a point of 
 allowing, indeed encouraging, sensible operator definition (*).
Unfortunately language design is often not as principled, scientific or ergonomic as it ought to be and is often heavily tainted by the moral opinions and personal practices of their designers. Unfortunately infrastructure matters more than ergonomics. I guess Python is a rare exception.
Jan 01
prev sibling next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Friday, 1 January 2016 at 11:37:40 UTC, Russel Winder wrote:
 (*) Arguably Scala takes this too far.
Btw, regarding operator overloading: Your comment about Scala made me think about the proposals for Algol which allowed the language to be heavily extended with even custom operator precedence IIRC. Interestingly, the Pony authors have decided that most operators should be explicitly grouped using parentheses, they argue that most programmers fail to remember precedence. I think that is mostly the case, especially with the somewhat flawed C precedence rules. Related note, I find this tongue-in-cheek comparison of Go and Algol68 is quite entertaining: http://cowlark.com/2009-11-15-go/ I think older languages such as Simula67, Algol68, Beta, ML etc still are quite interesting objects for study. It is scary how little progress have been made in the past 30 years... Fortunately some people are preserving books in PDF format. http://www.softwarepreservation.org/projects/ALGOL/book/Lindsey_van_der_Meulen-IItA68-Revised.pdf
Jan 02
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/1/2016 3:37 AM, Russel Winder via Digitalmars-d wrote:
 On Fri, 2016-01-01 at 10:45 +0000, Ola Fosheim Grøstad via Digitalmars-
 d wrote:
 In D1 Walter made a point about restricting operator overloading
 to discourage reuse of operators. In D2 there are many ways to
Java also went the route of "operator definition is too hard for programmers to deal with so we will not allow it".
"Also" is the wrong word. See my other post in this thread for why that was done, and no, it is not Java's reason. It has nothing to do with "too hard for programmers".
Jan 04
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2016-01-01 11:45, Ola Fosheim Grøstad wrote:

 In D1 Walter made a point about restricting operator overloading to
 discourage reuse of operators. In D2 there are many ways to create your
 own weird syntax, but Walter is locked on his D1 position, even though
 it makes little sense in D2.

 This is an overarching theme in D: the design rationale, that goes a
 decade back and is based on making a restricted easy to use version of
 C++, does not change, even though the context is very different in 2015
 and the premises has changed.

 These things are unlikely to change without a fork. Which is a pitty as
 D needs a more coherent design to get out of stagnation.
The core developers are making a big deal out of being able to have DSL's as string literals and process them at compile time. Although that's kind of pointless with SQL, since one still needs to send to string to the database to perform the query. The only thing that's possible is to validate the syntax at compile time. -- /Jacob Carlborg
Jan 02
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Saturday, 2 January 2016 at 12:23:30 UTC, Jacob Carlborg wrote:
 The core developers are making a big deal out of being able to 
 have DSL's as string literals and process them at compile time.
They have probably never done professional work with an ORM... Nobody wants that.
Jan 02
parent reply Jacob Carlborg <doob me.com> writes:
On 2016-01-02 13:55, Ola Fosheim Grøstad wrote:

 They have probably never done professional work with an ORM...

 Nobody wants that.
Exactly, but tell them ;) -- /Jacob Carlborg
Jan 02
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Saturday, 2 January 2016 at 19:46:48 UTC, Jacob Carlborg wrote:
 On 2016-01-02 13:55, Ola Fosheim Grøstad wrote:

 They have probably never done professional work with an ORM...

 Nobody wants that.
Exactly, but tell them ;)
I did: http://forum.dlang.org/post/mfdjcgykaizbzuuhoupv forum.dlang.org But Walter thinks that enforcing object-value comparisons in a rather limited way is a feature. It is a feature... but it is the wrong kind of feature... ;) It is an oversimplified and ineffective solution. A more sensible approach would be to use type-classes/concepts for defining what kind of comparison relation a class is required to provide and then deduce comparison operators based on what the programmer provides (e.g. if the programmer only provides a definition for "≤" then the compiler can deduce "<", ">", "≥", "==" etc.). D moved into templates and generics without cleaning up the language and is essentially ducktyped and macroish. Like C++. And Go (just in a different way). But C++ might start on a change process with C++17/20. We'll see.
Jan 02
prev sibling next sibling parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Saturday, 2 January 2016 at 12:23:30 UTC, Jacob Carlborg wrote:
 The core developers are making a big deal out of being able to 
 have DSL's as string literals and process them at compile time. 
 Although that's kind of pointless with SQL, since one still 
 needs to send to string to the database to perform the query. 
 The only thing that's possible is to validate the syntax at 
 compile time.
Well, you can also generate the structs and specific serialization code. And depending on how advanced your dsl you can also auto generate database migration code. There are propably tons of other stuff you can do with it. All in all much better than extending the language.
Jan 02
parent reply Jacob Carlborg <doob me.com> writes:
On 2016-01-02 19:37, Sebastiaan Koppe wrote:

 Well, you can also generate the structs and specific serialization code.
 And depending on how advanced your dsl you can also auto generate
 database migration code. There are propably tons of other stuff you can
 do with it.

 All in all much better than extending the language.
I would rather to the opposite. Generate the necessary SQL for a migration based on a struct, not the other way around. -- /Jacob Carlborg
Jan 02
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Saturday, 2 January 2016 at 19:48:26 UTC, Jacob Carlborg wrote:
 I would rather to the opposite. Generate the necessary SQL for 
 a migration based on a struct, not the other way around.
I meant that you generate the struct from the DSL, and then migration code from that struct.
Jan 02
parent reply Jacob Carlborg <doob me.com> writes:
On 2016-01-02 21:48, Sebastiaan Koppe wrote:

 I meant that you generate the struct from the DSL, and then migration
 code from that struct.
I don't think I understand, it seems complicated. -- /Jacob Carlborg
Jan 03
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Sunday, 3 January 2016 at 14:32:48 UTC, Jacob Carlborg wrote:
 On 2016-01-02 21:48, Sebastiaan Koppe wrote:

 I meant that you generate the struct from the DSL, and then 
 migration
 code from that struct.
I don't think I understand, it seems complicated.
Suppose you have this: mixin(db(` Entity Person Fields name -> string age -> integer Query byAge(a -> integer) -> age == a `)); which generates something like this: struct Person { string name; int age } auto getPersonByAge(DB db, int a) { return db.prepare!Person("SELECT name,age FROM Person WHERE age = ?").query(a); } and then later in time: mixin(db(` Entity Person Fields name -> string age -> integer phone -> string Query byAge(a -> integer) -> age == a `)); Given that you have access to both version it is easy to generate migration code for the phone field. Maybe it is contrived, but I think it shows you can do more with the DSL than just validating queries.
Jan 03
parent reply Jacob Carlborg <doob me.com> writes:
On 2016-01-03 17:45, Sebastiaan Koppe wrote:

 Suppose you have this:

 mixin(db(`
 Entity Person
    Fields
      name -> string
      age -> integer
    Query
      byAge(a -> integer) -> age == a
 `));

 which generates something like this:

 struct Person
 {
    string name;
    int age
 }
 auto getPersonByAge(DB db, int a)
 {
    return db.prepare!Person("SELECT name,age FROM Person WHERE age =
 ?").query(a);
 }

 and then later in time:

 mixin(db(`
 Entity Person
    Fields
      name -> string
      age -> integer
      phone -> string
    Query
      byAge(a -> integer) -> age == a
 `));

 Given that you have access to both version it is easy to generate
 migration code for the phone field.

 Maybe it is contrived, but I think it shows you can do more with the DSL
 than just validating queries.
Perhaps I'm missing something obvious but there are several problems with this: 1. What happens when you use more than one query for the same table at the same scope? In the above case, "Person" is already defined the second time "db" is invoked. It's not possible to add fields for already declared structs. Unless you use some form of opDispatch backed by an associative array of variants 2. What happens if I want to execute a custom query in a function, i.e. a query that is only used once. Will it generate the sturct inside the function or am I forced to always use this mixin at module level? I still think it's a lot easier to declare the struct with standard D code. I don't think the DSL adds any value in this case. Just do something like: db struct Person { string name; int age; } The db attribute would allow to create the migrations. -- /Jacob Carlborg
Jan 03
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Monday, 4 January 2016 at 07:48:14 UTC, Jacob Carlborg wrote:
 Perhaps I'm missing something obvious but there are several 
 problems with this:

 1. What happens when you use more than one query for the same 
 table at the same scope? In the above case, "Person" is already 
 defined the second time "db" is invoked. It's not possible to 
 add fields for already declared structs. Unless you use some 
 form of opDispatch backed by an associative array of variants
While it wasn't apparent, when I said "later" I meant "later" in git's history. The second version would be in a future commit, i.e. when changes are made. Then you can diff it and generate migration code. (Obviously this requires the DSL to be in a separate file without other code. We used to have such a reflective system in place in our continuous deployment. It would look at the changes in git since the last deployment and when certain files changed it knew it had to do certain stuff.)
 2. What happens if I want to execute a custom query in a 
 function, i.e. a query that is only used once. Will it generate 
 the sturct inside the function or am I forced to always use 
 this mixin at module level?
I suppose you need to declare each and every query in the DSL. But you might be on to something ugly in what I am proposing. Having said that, facebook seems to be going the DSL route. https://facebook.github.io/graphql and https://facebook.github.io/react/blog/2015/05/01/graphql-introduction.html Another benefit of a DSL is that you can generate code for other languages. For instance you could generate a JS, Swift or Android client.
 I still think it's a lot easier to declare the struct with 
 standard D code. I don't think the DSL adds any value in this 
 case. Just do something like:

  db struct Person
 {
     string name;
     int age;
 }

 The  db attribute would allow to create the migrations.
But then you are back to square one with regards to queries.
Jan 04
parent reply Jacob Carlborg <doob me.com> writes:
On 2016-01-04 12:14, Sebastiaan Koppe wrote:

 But then you are back to square one with regards to queries.
If you can't use the host language, i.e. Person.where(e => e.name == "John");, what's the point of inventing a new DSL? Why not just use SQL if the DSL needs to be in a string anyway. -- /Jacob Carlborg
Jan 04
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 01/04/2016 07:39 AM, Jacob Carlborg wrote:
 If you can't use the host language, i.e. Person.where(e => e.name ==
 "John");, what's the point of inventing a new DSL? Why not just use SQL
 if the DSL needs to be in a string anyway.
I agree. A CTFE SQL parser with automatic variable binding would be pretty rad. Not to mention it would by design take care of things like string injection attacks etc. Also: finding SQL syntax errors during compilation of D code... priceless. -- Andrei
Jan 04
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/2/16 7:23 AM, Jacob Carlborg wrote:
 On 2016-01-01 11:45, Ola Fosheim Grøstad wrote:

 In D1 Walter made a point about restricting operator overloading to
 discourage reuse of operators. In D2 there are many ways to create your
 own weird syntax, but Walter is locked on his D1 position, even though
 it makes little sense in D2.

 This is an overarching theme in D: the design rationale, that goes a
 decade back and is based on making a restricted easy to use version of
 C++, does not change, even though the context is very different in 2015
 and the premises has changed.

 These things are unlikely to change without a fork. Which is a pitty as
 D needs a more coherent design to get out of stagnation.
The core developers are making a big deal out of being able to have DSL's as string literals and process them at compile time. Although that's kind of pointless with SQL, since one still needs to send to string to the database to perform the query. The only thing that's possible is to validate the syntax at compile time.
Binding D variables to SQL expressions also comes to mind. -- Andrei
Jan 03
parent Jacob Carlborg <doob me.com> writes:
On 2016-01-04 01:50, Andrei Alexandrescu wrote:

 Binding D variables to SQL expressions also comes to mind. -- Andrei
You have also been pushing a lot for ranges, which I think is good. I would much rather like to view a table as a range, but the algorithms would be preform in the database instead of in the application code. Something like: Person.filter!(e => e.name == "John") -- /Jacob Carlborg
Jan 03
prev sibling parent reply Piotrek <nodata nodata.pl> writes:
On Friday, 1 January 2016 at 10:00:43 UTC, Kapps wrote:

 This example shows the difficulty of doing this in D. You can't 
 really have something like `p.Name == "James"`, or `p.Age < 21` 
 translate to SQL properly without language changes, which I 
 believe Walter or Andrei were against. This has been the key 
 problem when things like Linq to Sql for D have been brought up 
 before.
Not really. There is no translation stage to SQL or any other DSL in the proposal. So this problem doesn't exist and no language changes are needed. However there is another issue which must be taken into account. One should decide if the object is retrieved directly or via via proxy. Especially for big objects with lot of aggregated types. Piotrek
Jan 02
parent reply Chris Wright <dhasenan gmail.com> writes:
On Sat, 02 Jan 2016 16:40:16 +0000, Piotrek wrote:

 On Friday, 1 January 2016 at 10:00:43 UTC, Kapps wrote:
 
 This example shows the difficulty of doing this in D. You can't really
 have something like `p.Name == "James"`, or `p.Age < 21`
 translate to SQL properly without language changes, which I believe
 Walter or Andrei were against. This has been the key problem when
 things like Linq to Sql for D have been brought up before.
Not really. There is no translation stage to SQL or any other DSL in the proposal. So this problem doesn't exist and no language changes are needed.
So you want to create the following query: people.filter!(x => x.surname == "Slughorn"); And you've got ten million people in the collection, and you want this query to finish soonish. So you need to use an index. But a full index scan isn't so great; you want to do an index lookup if possible. That's simple enough; we generate proxy types to record what properties you're using and what operations you're performing. PersonProxy records that you're accessing a field 'surname', gives a StringFieldProxy, and that records that you're checking for equality with the string "Slughorn". The lambda returns true when opEquals returns true. But people write queries that are more complex than that, like: people.filter!(x => x.surname == "Slughorn" || x.age <= 17); First time you run this, x.surname.opEquals("Slughorn") returns true and the expression as a whole returns true. You missed the second part of the expression. That's bad. So we need to evaluate this lambda twice per parameter. (Actually, thanks to opCmp, sometimes you'll have to evaluate it three times.) We use that to build up a giant truth table, and then we can execute your query. And that "twice per parameter" thing is exponential, and we build up a truth table that's exponentially large with respect to the complexity of the query. Some queries I've written for production systems would take a week for this system to prepare to execute and require a petabyte of storage space. This is, shall we say, less than ideal. You might be able to support queries as large as ten comparisons in a reasonable timeframe. But for all but the most trivial queries, it'll be faster to use SQL. Fortunately, trivial queries are common. You could write this system and have it work up to, say, five comparisons, and when your query exceeds that limit, it will throw an exception asking you to rewrite the query in SQL. It wouldn't be able to tell you what the equivalent query is, however. It's using a truth table and doesn't have access to what you wrote. I mean, sure, it could try to work backwards from the truth table, but that's rather expensive.
Jan 02
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/2/16 3:47 PM, Chris Wright wrote:
 On Sat, 02 Jan 2016 16:40:16 +0000, Piotrek wrote:

 On Friday, 1 January 2016 at 10:00:43 UTC, Kapps wrote:

 This example shows the difficulty of doing this in D. You can't really
 have something like `p.Name == "James"`, or `p.Age < 21`
 translate to SQL properly without language changes, which I believe
 Walter or Andrei were against. This has been the key problem when
 things like Linq to Sql for D have been brought up before.
Not really. There is no translation stage to SQL or any other DSL in the proposal. So this problem doesn't exist and no language changes are needed.
So you want to create the following query: people.filter!(x => x.surname == "Slughorn"); And you've got ten million people in the collection, and you want this query to finish soonish. So you need to use an index. But a full index scan isn't so great; you want to do an index lookup if possible. That's simple enough; we generate proxy types to record what properties you're using and what operations you're performing. PersonProxy records that you're accessing a field 'surname', gives a StringFieldProxy, and that records that you're checking for equality with the string "Slughorn". The lambda returns true when opEquals returns true. But people write queries that are more complex than that, like: people.filter!(x => x.surname == "Slughorn" || x.age <= 17); First time you run this, x.surname.opEquals("Slughorn") returns true and the expression as a whole returns true. You missed the second part of the expression. That's bad. So we need to evaluate this lambda twice per parameter. (Actually, thanks to opCmp, sometimes you'll have to evaluate it three times.) We use that to build up a giant truth table, and then we can execute your query. And that "twice per parameter" thing is exponential, and we build up a truth table that's exponentially large with respect to the complexity of the query. Some queries I've written for production systems would take a week for this system to prepare to execute and require a petabyte of storage space. This is, shall we say, less than ideal.
This may in fact be good signal that an approach based on expression templates is not the most appropriate for D. -- Andrei
Jan 03
parent reply Jacob Carlborg <doob me.com> writes:
On 2016-01-04 00:50, Andrei Alexandrescu wrote:

 This may in fact be good signal that an approach based on expression
 templates is not the most appropriate for D. -- Andrei
This whole thread has already discussed and showed that D operator overloading is lacking in terms of expression templates. The post by Chris assumes no language changes. If D supported overloading all operators the opEquals method would not return true, it would return a proxy that overloaded the || operator. The whole lambda expression would only generate a single query. The rest of the post falls apart from this mistake. -- /Jacob Carlborg
Jan 03
parent Piotrek <nodata nodata.pl> writes:
On Monday, 4 January 2016 at 07:59:40 UTC, Jacob Carlborg wrote:
 On 2016-01-04 00:50, Andrei Alexandrescu wrote:

 This may in fact be good signal that an approach based on 
 expression
 templates is not the most appropriate for D. -- Andrei
This whole thread has already discussed and showed that D operator overloading is lacking in terms of expression templates. The post by Chris assumes no language changes. If D supported overloading all operators the opEquals method would not return true, it would return a proxy that overloaded the || operator. The whole lambda expression would only generate a single query. The rest of the post falls apart from this mistake.
But most of the posts are related to DSL (->SQL). What do you think about solution where no DSL, VM or other intermediate layer is used? Also I have a theory that SQL appeared because of limiting expressiveness of other programming languages (like C) ;) Cheers Piotrek
Jan 04
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2016-01-02 21:47, Chris Wright wrote:

 So you want to create the following query:

    people.filter!(x => x.surname == "Slughorn");

 And you've got ten million people in the collection, and you want this
 query to finish soonish. So you need to use an index. But a full index
 scan isn't so great; you want to do an index lookup if possible.

 That's simple enough; we generate proxy types to record what properties
 you're using and what operations you're performing. PersonProxy records
 that you're accessing a field 'surname', gives a StringFieldProxy, and
 that records that you're checking for equality with the string "Slughorn".
 The lambda returns true when opEquals returns true.

 But people write queries that are more complex than that, like:

    people.filter!(x => x.surname == "Slughorn" || x.age <= 17);

 First time you run this, x.surname.opEquals("Slughorn") returns true and
 the expression as a whole returns true. You missed the second part of the
 expression. That's bad.
If D had better operator overloading or supported AST macros, opEquals would not return true. It would return a new proxy that overloads the || operator. The rest of the post falls apart from this mistake. -- /Jacob Carlborg
Jan 03
next sibling parent reply rsw0x <anonymous anonymous.com> writes:
On Monday, 4 January 2016 at 07:55:53 UTC, Jacob Carlborg wrote:
 On 2016-01-02 21:47, Chris Wright wrote:

 So you want to create the following query:

    people.filter!(x => x.surname == "Slughorn");

 And you've got ten million people in the collection, and you 
 want this
 query to finish soonish. So you need to use an index. But a 
 full index
 scan isn't so great; you want to do an index lookup if 
 possible.

 That's simple enough; we generate proxy types to record what 
 properties
 you're using and what operations you're performing. 
 PersonProxy records
 that you're accessing a field 'surname', gives a 
 StringFieldProxy, and
 that records that you're checking for equality with the string 
 "Slughorn".
 The lambda returns true when opEquals returns true.

 But people write queries that are more complex than that, like:

    people.filter!(x => x.surname == "Slughorn" || x.age <= 17);

 First time you run this, x.surname.opEquals("Slughorn") 
 returns true and
 the expression as a whole returns true. You missed the second 
 part of the
 expression. That's bad.
If D had better operator overloading or supported AST macros, opEquals would not return true. It would return a new proxy that overloads the || operator. The rest of the post falls apart from this mistake.
compiler plugins like Rust to enable AST macros could fix this but walter and andrei seem extremely opposed to any form of macros so we end up with ugly heavily templated ugly band-aids shame, because I think D will quickly lose their lead in the metaprogramming area if Rust keeps it up - e.g, someone already made a GC Plugin for rust's compiler
Jan 04
parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 2016-01-04 at 08:29 +0000, rsw0x via Digitalmars-d wrote:
=20
[=E2=80=A6]
 compiler plugins like Rust to enable AST macros could fix this=C2=A0
 but walter and andrei seem extremely opposed to any form of=C2=A0
 macros so we end up with ugly heavily templated ugly band-aids
Rust may have macros, in fact it has two sorts, but they are not as useful as you might at first think. The macro system as in Rust stable is not up to the task of doing code generation such that you can replicate PyTest's pytest.mark.parametrize decorator in Rust. The compiler plugins that would allow this are only allowed in Rust nightly and are forbidden in =C2=A0Rust beta and stable. Rust nightly is effectively a totally different platform to Rust beta or stable, which is undermining the Rust platform. =C2=A0 I must now try creating a D version of the pytest.mark.parametrize decorator =E2=80=93 unless someone already has and I have just missed it.
 shame, because I think D will quickly lose their lead in the=C2=A0
 metaprogramming area if Rust keeps it up - e.g, someone already=C2=A0
 made a GC Plugin for rust's compiler
I think Rust and not C++17 is the biggest threat to D just now. I am still intending to rewrite Me TV in D. --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Jan 04
parent reply John Colvin <john.loughran.colvin gmail.com> writes:
On Monday, 4 January 2016 at 12:28:47 UTC, Russel Winder wrote:
 I must now try creating a D version of the 
 pytest.mark.parametrize decorator – unless someone already has 
 and I have just missed it.
I quick look at pytest.mark.parametrize suggests it could be implemented with UDAs and a test-runner that finds all declarations a module (recursively) and does all the relevant logic (e.g. that's got more than one instance of parametrize, so do some sort of cartesian product of the inputs) and actually runs the test. The main thing that python has here over D is that D's UDAs can't directly modify the function they're attached to, but I don't think that's necessary for parametrize. Interestingly, functions can query their own attributes: (3) auto attr() { return __traits(getAttributes, attr)[0]; } unittest { assert(attr() == 3); } not sure when I'd use that though...
Jan 04
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 01/04/2016 08:09 AM, John Colvin wrote:
 On Monday, 4 January 2016 at 12:28:47 UTC, Russel Winder wrote:
 I must now try creating a D version of the pytest.mark.parametrize
 decorator – unless someone already has and I have just missed it.
I quick look at pytest.mark.parametrize suggests it could be implemented with UDAs and a test-runner that finds all declarations a module (recursively) and does all the relevant logic (e.g. that's got more than one instance of parametrize, so do some sort of cartesian product of the inputs) and actually runs the test. The main thing that python has here over D is that D's UDAs can't directly modify the function they're attached to, but I don't think that's necessary for parametrize. Interestingly, functions can query their own attributes: (3) auto attr() { return __traits(getAttributes, attr)[0]; } unittest { assert(attr() == 3); } not sure when I'd use that though...
For computed attributes in template functions. -- Andrei
Jan 04
prev sibling parent Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Mon, 2016-01-04 at 13:09 +0000, John Colvin via Digitalmars-d wrote:
=20
[=E2=80=A6]
 not sure when I'd use that though...
At a trivial level that nonetheless leads to functionality that can be used more sensibly, think of testing factorial. You want property-based=20 testing, but also you want a (well usually two) table of examples. Python via pytest.mark.parametrize does the latter very well. Hypothesis does the former. Spock enables both very easily. It is know that the tests for the Java driver to MongoDB relies on table-driven and property-based testing, mostly using Spock. --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Jan 04
prev sibling parent reply Chris Wright <dhasenan gmail.com> writes:
On Mon, 04 Jan 2016 08:55:53 +0100, Jacob Carlborg wrote:

 If D had better operator overloading or supported AST macros, opEquals
 would not return true. It would return a new proxy that overloads the ||
 operator. The rest of the post falls apart from this mistake.
Not proposing language changes was an intentional feature, not a mistake.
Jan 04
parent reply Jacob Carlborg <doob me.com> writes:
On 2016-01-05 05:16, Chris Wright wrote:

 Not proposing language changes was an intentional feature, not a mistake.
Then you obviously can't use the operators. You would have to fall back to methods: Person.where!(e => e.name.eq("John")); -- /Jacob Carlborg
Jan 05
next sibling parent reply Chris Wright <dhasenan gmail.com> writes:
On Tue, 05 Jan 2016 09:26:16 +0100, Jacob Carlborg wrote:

 On 2016-01-05 05:16, Chris Wright wrote:
 
 Not proposing language changes was an intentional feature, not a
 mistake.
Then you obviously can't use the operators. You would have to fall back to methods: Person.where!(e => e.name.eq("John"));
Which some people call a DSL, and DSLs were, I understand, something we were trying to avoid. You get far less readability than you do with plain D or a SQL string. Plus side, you spend no time parsing, but it took less than a millisecond to parse your SQL query in the first place, and most SQL databases allow you to cache prepared statements.
Jan 05
parent Jacob Carlborg <doob me.com> writes:
On 2016-01-05 18:32, Chris Wright wrote:

 Which some people call a DSL, and DSLs were, I understand, something we
 were trying to avoid. You get far less readability than you do with plain
 D or a SQL string.
The above is plain D. Not sure what you're referring to. And a DSL is usually used to increase readability. Andrei and Walter want to avoid expression templates, not a DSL, as far as I understand it. -- /Jacob Carlborg
Jan 06
prev sibling parent reply Mengu <mengukagan gmail.com> writes:
On Tuesday, 5 January 2016 at 08:26:16 UTC, Jacob Carlborg wrote:
 On 2016-01-05 05:16, Chris Wright wrote:

 Not proposing language changes was an intentional feature, not 
 a mistake.
Then you obviously can't use the operators. You would have to fall back to methods: Person.where!(e => e.name.eq("John"));
i love how things can become so complex in this community. i am a web developer so i am going to talk in terms of simplicity. as russel said, with SQLAlchemy (python library) operators are overloaded and we can do DBSession.query(User).filter(User.first_name == "Jacob", User.age
 27). this will generate the following query:
"SELECT * FROM users u WHERE u.first_name = 'Jacob' AND u.age > 27" and will let the DB handle the rest. but as a web developer, i don't mind if i have to type User.where("first_name = ? AND age > 27", request.POST.get('name')). this will generate the same query above. i added the question mark to say the parameters should be escaped properly. guys, what we need is a DB abstraction supporting PostgreSQL, MySQL and other major database systems and we need it yesterday. let's not make things more complex than they are and build up this thing. p.s. this is not a reply to jacob, just his post was the one i clicked reply for. :-)
Jan 05
next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Tuesday, 5 January 2016 at 22:12:05 UTC, Mengu wrote:
 guys, what we need is a DB abstraction supporting PostgreSQL, 
 MySQL and other major database systems and we need it 
 yesterday. let's not make things more complex than they are and 
 build up this thing.
What is needed isn't a DB abstraction, but a language that makes elegant programming abstractions easy to implement for a wide variety of problem areas. You can do DB abstractions in any language... No point in having a thread about it if anything goes. Just do 2-3 different prototypes and select the best ones. No discussion needed...
Jan 05
prev sibling next sibling parent Marc =?UTF-8?B?U2Now7x0eg==?= <schuetzm gmx.net> writes:
On Tuesday, 5 January 2016 at 22:12:05 UTC, Mengu wrote:
 but as a web developer, i don't mind if i have to type 
 User.where("first_name = ? AND age > 27", 
 request.POST.get('name')). this will generate the same query 
 above. i added the question mark to say the parameters should 
 be escaped properly.
As a web developer, I _do_ care, though. For one, I shouldn't have to know how the fields are called in the database. But more importantly, it isn't composable: in the general case, you can not write e.g. User.joins("other_table ON ...").where("first_name = ? ..."); and expect it to work, because `first_name` could be ambiguous. This in turn means that you can't create abstractions, for example: auto users = User .onlyMale .where(["birthday": Date.today]) .orderBy!(User.firstName); Here your query would have to know how male and female users are represented in your database, e.g. by an integer, a lowercase 'm', an uppercase 'M'... Such lack of abstraction can quickly make your codebase unmaintainable.
Jan 06
prev sibling next sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 5 January 2016 at 22:12:05 UTC, Mengu wrote:
 guys, what we need is a DB abstraction supporting PostgreSQL, 
 MySQL and other major database systems and we need it 
 yesterday. let's not make things more complex than they are and 
 build up this thing.
I'm personally a fan of dplyr using R. bachmeier is apparently working on a way to embed R in D. Might be an option if you need it yesterday. An example I saw above of people.filter!(x => x.surname == "Slughorn"); looked a lot like a D version of dplyr's filter function.
Jan 07
prev sibling parent reply Mengu <mengukagan gmail.com> writes:
On Tuesday, 5 January 2016 at 22:12:05 UTC, Mengu wrote:
 On Tuesday, 5 January 2016 at 08:26:16 UTC, Jacob Carlborg 
 wrote:
 [...]
i love how things can become so complex in this community. i am a web developer so i am going to talk in terms of simplicity. as russel said, with SQLAlchemy (python library) operators are overloaded and we can do DBSession.query(User).filter(User.first_name == "Jacob", User.age > 27). this will generate the following query: "SELECT * FROM users u WHERE u.first_name = 'Jacob' AND u.age > 27" and will let the DB handle the rest. but as a web developer, i don't mind if i have to type User.where("first_name = ? AND age > 27", request.POST.get('name')). this will generate the same query above. i added the question mark to say the parameters should be escaped properly. guys, what we need is a DB abstraction supporting PostgreSQL, MySQL and other major database systems and we need it yesterday. let's not make things more complex than they are and build up this thing. p.s. this is not a reply to jacob, just his post was the one i clicked reply for. :-)
and while we were talking the talk, rust community rolled out something good called diesel. check it out at http://diesel.rs/. we need tools that get things done. we do not need tools that makes things more complex than they already are.
Feb 05
next sibling parent Sebastiaan Koppe <mail skoppe.eu> writes:
On Saturday, 6 February 2016 at 00:14:08 UTC, Mengu wrote:
 and while we were talking the talk, rust community rolled out 
 something good called diesel. check it out at http://diesel.rs/.

 we need tools that get things done. we do not need tools that 
 makes things more complex than they already are.
I saw the examples on the site. In every case I prefer reading SQL instead of a complicated function chain with semantics I have yet to learn; SQL I already know. A nice thing is the `less boilerplate` example. But I think they took a wrong approach by deriving the class from Queryable. I like D's approach better.
Feb 06
prev sibling parent reply Piotrek <nodata nodata.pl> writes:
On Saturday, 6 February 2016 at 00:14:08 UTC, Mengu wrote:
 and while we were talking the talk, rust community rolled out 
 something good called diesel. check it out at http://diesel.rs/.

 we need tools that get things done. we do not need tools that 
 makes things more complex than they already are.
Almost no one (including me) is interested in ORM for SQL. The point is ORM+SQL is limiting and sooner or later you fallback to SQL. Additionally there is no critical mass for this kind of big project (combining all the SQL engines - good luck). Andrei suggested a CTFE sql parser, so people who like SQL (not me) can benefit from the D metaprogramming power. For the rest there is my proposal ;) : a language embedded DB. As far as I can tell none of the known PLes has this "killer" feature. Piotrek
Feb 06
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Saturday, 6 February 2016 at 13:41:03 UTC, Piotrek wrote:
 On Saturday, 6 February 2016 at 00:14:08 UTC, Mengu wrote:
 and while we were talking the talk, rust community rolled out 
 something good called diesel. check it out at 
 http://diesel.rs/.

 we need tools that get things done. we do not need tools that 
 makes things more complex than they already are.
Almost no one (including me) is interested in ORM for SQL. The point is ORM+SQL is limiting and sooner or later you fallback to SQL.
A good ORM-like interface is mandatory for working with NoSQL databases...
Feb 06
parent Piotrek <nodata nodata.pl> writes:
On Saturday, 6 February 2016 at 14:04:42 UTC, Ola Fosheim Grøstad 
wrote:
 A good ORM-like interface is mandatory for working with NoSQL 
 databases...
Fortunately, I don't plan to work with so called NoSQL databases... BTW. Take a look at the example from the PoC code and check what works currently (i.e. passing unit tests) https://github.com/PiotrekDlang/AirLock/blob/master/docs/database/design.md#example Piotrek
Feb 06
prev sibling next sibling parent Charles <csmith.ku2013 gmail.com> writes:
On Saturday, 6 February 2016 at 13:41:03 UTC, Piotrek wrote:
 For the rest there is my proposal ;) : a language embedded DB. 
 As far as I can tell none of the known PLes has this "killer" 
 feature.

 Piotrek
SAS does, and has for quite a few decades now. Its a pretty big corporate language used for statistics and analysis. It does work exceptionally well, easily handling well over 100M records within a single table. They implement a table per file (sas7bdat). That said, the binary table format is proprietary, and while there have been attempts to reverse engineer it none have been completely successful.
Feb 06
prev sibling next sibling parent Chris Wright <dhasenan gmail.com> writes:
On Sat, 06 Feb 2016 13:41:03 +0000, Piotrek wrote:

 On Saturday, 6 February 2016 at 00:14:08 UTC, Mengu wrote:
 and while we were talking the talk, rust community rolled out something
 good called diesel. check it out at http://diesel.rs/.

 we need tools that get things done. we do not need tools that makes
 things more complex than they already are.
Almost no one (including me) is interested in ORM for SQL.
You mean, a high-level query language abstracting across multiple databases, expressed in terms of my object model rather than physical tables. You're talking about HQL, not Hibernate as a whole. Nobody wants to write manual serialization and deserialization code. Not when there's a reliable, low-cost alternative. And if I could just write my object model and have an automated system create my database tables and upgrade scripts for me, that would be cool too.
 The point is
 ORM+SQL is limiting and sooner or later you fallback to SQL.
The HQL I recall would be bad for reporting and aggregation, but since I never needed that, perhaps I just didn't find the relevant options and syntax. I find HQL slightly nicer to work with than SQL, but since the databases themselves aren't speaking it, it's not so handy. I found one query that I needed where HQL could do it in two minutes and hand-written SQL could do it in fifty milliseconds. That's rare, but it does happen.
Feb 06
prev sibling next sibling parent David DeWitt <dkdewitt gmail.com> writes:
On Saturday, 6 February 2016 at 13:41:03 UTC, Piotrek wrote:
 On Saturday, 6 February 2016 at 00:14:08 UTC, Mengu wrote:
 and while we were talking the talk, rust community rolled out 
 something good called diesel. check it out at 
 http://diesel.rs/.

 we need tools that get things done. we do not need tools that 
 makes things more complex than they already are.
Almost no one (including me) is interested in ORM for SQL. The point is ORM+SQL is limiting and sooner or later you fallback to SQL.
I wouldn't say almost no one. I know Sql Alchemy is very popular and used by a lot of ppl. Ive used and not used ORMs on numerous projects and sometimes they make sense and sometimes they do not. I agree its not needed but alot of these tools are used by average developers and not having them isn't going to help D. Not that we don't need them but when ppl say "Well I like D but find there are missing libraries I want" A decent ORM would prolly be one of them. Basically if Rust continues to get these libraries and they are decent then they will def attract more ppl to the language instead of them possibly coming to D if that is important which I know has been brought up before and I think projects and tools like these with further attract ppl.
Feb 06
prev sibling parent reply Mengu <mengukagan gmail.com> writes:
On Saturday, 6 February 2016 at 13:41:03 UTC, Piotrek wrote:
 On Saturday, 6 February 2016 at 00:14:08 UTC, Mengu wrote:
 and while we were talking the talk, rust community rolled out 
 something good called diesel. check it out at 
 http://diesel.rs/.

 we need tools that get things done. we do not need tools that 
 makes things more complex than they already are.
Almost no one (including me) is interested in ORM for SQL. The point is ORM+SQL is limiting and sooner or later you fallback to SQL. Additionally there is no critical mass for this kind of big project (combining all the SQL engines - good luck). Andrei suggested a CTFE sql parser, so people who like SQL (not me) can benefit from the D metaprogramming power. For the rest there is my proposal ;) : a language embedded DB. As far as I can tell none of the known PLes has this "killer" feature. Piotrek
i don't mind if it's an ORM or something else. my point was that instead of complaining about stuff, we need a safe, stable and extendable database library supporting sqlite, mysql, postgresql, mssql and oracle dbs and we need it like yesterday. nothing fancy. people can get creative and fancy over that standard api and users get to choose.
Feb 08
parent Zardoz <luis.panadero gmail.com> writes:
On Monday, 8 February 2016 at 13:36:09 UTC, Mengu wrote:
 i don't mind if it's an ORM or something else. my point was 
 that instead of complaining about stuff, we need a safe, stable 
 and extendable database library supporting sqlite, mysql, 
 postgresql, mssql and oracle dbs and we need it like yesterday. 
 nothing fancy. people can get creative and fancy over that 
 standard api and users get to choose.
Something like JDBC on D ... --> https://code.dlang.org/packages/ddbc
Feb 09
prev sibling parent reply Piotrek <nodata nodata.pl> writes:
On Saturday, 2 January 2016 at 20:47:37 UTC, Chris Wright wrote:
 So you want to create the following query:

   people.filter!(x => x.surname == "Slughorn");

 And you've got ten million people in the collection, and you 
 want this query to finish soonish. So you need to use an index. 
 But a full index scan isn't so great; you want to do an index 
 lookup if possible.
Correct.
 That's simple enough; we generate proxy types to record what 
 properties you're using and what operations you're performing. 
 PersonProxy records that you're accessing a field 'surname', 
 gives a StringFieldProxy, and that records that you're checking 
 for equality with the string "Slughorn". The lambda returns 
 true when opEquals returns true.

 But people write queries that are more complex than that, like:

   people.filter!(x => x.surname == "Slughorn" || x.age <= 17);

 First time you run this, x.surname.opEquals("Slughorn") returns 
 true and the expression as a whole returns true. You missed the 
 second part of the expression. That's bad.
 So we need to evaluate this lambda twice per parameter.
Hmm. Probably we don't think about the "proxy" term in the same way. When I mentioned proxy I thought about a skeleton object to access parts of the original heavy object. So in fact, filter works on proxy and one evaluation is performed no matter how complex the condition is. The proxy is responsible to retrieve needed data. Additionally (I haven't mentioned it yet) we can provide mechanism for data integrity when joining separate objects.
 You might be able to support queries as large as ten 
 comparisons in a reasonable timeframe. But for all but the most 
 trivial queries, it'll be faster to use SQL.
SQL is no magic. You can write a code to beat any SQL planner. Also I think you had "joins" in mind which I guess are the biggest problem for performance. But "What if I tell there is no SQL" !!!! Not even under the hood. Piotrek
Jan 04
parent reply Chris Wright <dhasenan gmail.com> writes:
On Mon, 04 Jan 2016 16:59:47 +0000, Piotrek wrote:
 SQL is no magic.
True. You could equivalently have a string containing valid D code, accompanied by CTFE parsers that will determine which indices to use. This has typically been considered an antipattern. It tends to work poorly with refactoring tools, for instance, and there's an assumption that string literals are data and not code. C#'s system of lambda expressions that you can access as expression trees keeps the benefits of writing everything in the same language.
Jan 04
parent Piotrek <nodata nodata.pl> writes:
On Tuesday, 5 January 2016 at 04:19:01 UTC, Chris Wright wrote:
 You could equivalently have a string containing valid D code, 
 accompanied by CTFE parsers that will determine which indices 
 to use. This has typically been considered an antipattern. It 
 tends to work poorly with refactoring tools, for instance, and 
 there's an assumption that string literals are data and not 
 code.
Would the D string tokens do? BTW. Thanks for all your valuable comments. I really like that kind of critique.
 C#'s system of lambda expressions that you can access as 
 expression trees keeps the benefits of writing everything in 
 the same language.
I think I'm still not convinced to that approach (seems hacky). I agree with Walter in that matter (language changes, AST, overloading etc). Especially, D is number one in meta-programming capabilities (among C++ and Rust) and nothing seems to change it in foreseeable future (https://users.rust-lang.org/t/rust-vs-dlang-i-want-more-experienced/4472/11) Piotrek
Feb 06
prev sibling next sibling parent reply Abdulhaq <alynch4047 gmail.com> writes:
On Thursday, 31 December 2015 at 17:14:55 UTC, Piotrek wrote:
 The goal of this post is to measure the craziness of an idea to 
 embed a database engine into the D language ;)

 I think about a database engine which would meet my three main 
 requirements:
   - integrated with D (ranges)
   - ACID
   - fast

 Since the days when I was working on financing data SW I become 
 allergic to SQL. I though that NoSQL databases would fill the 
 bill. Unfortunately they didn't. And I want to have an ability 
 to write a code like this without too much effort:

   struct Person
   {
    string name;
    string surname;
    ubyte age;
    Address address;
   }

  DataBase db = new DataBase("file.db");
  auto coll = db.collection!Person("NSA.Registry");
  auto visitationList = coll.filter!(p => p.name == "James");
  writeln (visitationList);

 And other things like updating and deleting from db. I think 
 you get my point.

 So I started a PoC project based on SQLite design:
 https://github.com/PiotrekDlang/AirLock/blob/master/docs/database/design.md#architecture

 The PoC code: 
 https://github.com/PiotrekDlang/AirLock/tree/master/src/database

 Can you please share your thoughts and experience on the topic? 
 Has anyone tried similar things?

 Piotrek
My two pence, if you want it to be fast then it must have a good implementation of indices. Your filter functions should not actually start collecting real records, but instead should simply change the way that the cursor traverses the underlying data store. You will need good query 'compilation' like the big boys do, which work out which tables and indices to use and in which order, based on stats of the data / indices. If you want ACID then SQL seems like a good approach to me, certainly I wouldn't want anything ORM-like for updating / inserting data. There a number of good libraries out there already, SQLite obviously springs to mind. It would be a fun project but perhaps a lot more work than you realised if you really want isolation levels, speed etc.
Jan 03
parent Piotrek <nodata nodata.pl> writes:
On Sunday, 3 January 2016 at 19:48:42 UTC, Abdulhaq wrote:
 My two pence, if you want it to be fast then it must have a 
 good implementation of indices. Your filter functions should 
 not actually start collecting real records, but instead should 
 simply change the way that the cursor traverses the underlying 
 data store. You will need good query 'compilation' like the big 
 boys do, which work out which tables and indices to use and in 
 which order, based on stats of the data / indices.
Agree (I mentioned about proxies before). But I'm not sure about "good query compilation" thing. Good storage management should do. Rest can be done in user code.
 If you want ACID then SQL seems like a good approach to me, 
 certainly I wouldn't want anything ORM-like for updating / 
 inserting data.
Well. Some people really like SQL. I'm not one of them. I'm neither a fan of ORM+SQL. But I think ACID is not only reserved for SQL.
 There a number of good libraries out there already, SQLite 
 obviously springs to mind.
The SQLite project is great.
 It would be a fun project but perhaps a lot more work than you 
 realised if you really want isolation levels, speed etc.
Yes, I know. But one step at a time. It's a some kind of fun for me as well ;) Cheers Piotrek
Jan 04
prev sibling next sibling parent reply Jakob Jenkov <jakob jenkov.com> writes:
You could just target your database at data analysis. Then you 
don't need to care about ACID, transactions etc. Just load all 
the data into memory, and start analyzing it.

Also, you'd typically be scanning over large parts of the data 
set for each query, so you may not need to support a full query 
language. Just what is needed for data analysis.

Later you can modify your engine to support ACID, more expressive 
query language etc.

On one of the projects I am working on right now, we will also 
implement our own database engine. We need it to integrate 
tightly with the rest our architecture, and the only way to do 
that is to roll our own. We will also not be using SQL because 
SQL is so limiting.

So, I'd say "go ahead" - you can only learn something from the 
project. I've "reinvented a lot of wheels" over the years, and 
each time I came out smarter than before. Not every reinvention 
was a success, but I always learned something from the process.
Jan 03
parent reply Piotrek <nodata nodata.pl> writes:
On Sunday, 3 January 2016 at 23:22:17 UTC, Jakob Jenkov wrote:
 You could just target your database at data analysis. Then you 
 don't need to care about ACID, transactions etc. Just load all 
 the data into memory, and start analyzing it.

 Also, you'd typically be scanning over large parts of the data 
 set for each query, so you may not need to support a full query 
 language. Just what is needed for data analysis.

 Later you can modify your engine to support ACID, more 
 expressive query language etc.
That's the plan:) Except no dedicated query language is planned. At least that's my vision based on what I know about D and databases currently.
 On one of the projects I am working on right now, we will also 
 implement our own database engine. We need it to integrate 
 tightly with the rest our architecture, and the only way to do 
 that is to roll our own. We will also not be using SQL because 
 SQL is so limiting.

 So, I'd say "go ahead" - you can only learn something from the 
 project. I've "reinvented a lot of wheels" over the years, and 
 each time I came out smarter than before. Not every reinvention 
 was a success, but I always learned something from the process.
Thanks! So at least one more soul believing that D can approach the SQL expressiveness in db domain. Cheers Piotrek
Jan 04
parent Gianni Pisetta <pisetta.gianni alice.it> writes:
On Monday, 4 January 2016 at 17:36:20 UTC, Piotrek wrote:
 Thanks! So at least one more soul believing that D can approach 
 the SQL expressiveness in db domain.

 Cheers
 Piotrek
Some time ago I had the same idea as yours. IMHO, no D to SQL is really needed for this, and the same goes for relational databases. My idea was to initially build a DB as library, using graph databases as starting point, where the metadata is defined only one time in the code. The query methods must allow easy access not only to the data, but also to the relations defined from object to object. As a starting point you must be able to do something like this: // Stores are where the data is saved auto localStore = FileStore!Endian.littleEndian( "some_file_name" ); // Local file // load() and save() on objects of type User uses this store. User.setDefaultStore( localStore ); User user = new User(); user.name = "John"; user.password = sha1( "password" ); // Uses the default store auto localId = user.save(); // Uses the store specified as argument auto otherId = user.save( someOtherStore ); // load the object from the default store User localUser = User.load( localId ); // load the object from the network store User otherUser = User.load( someOtherStore, otherId ); // Ideally users where the last login is older than one year using the networkStore auto oldUsers = User.all( someOtherStore ) .filterUsing!(User.lastLogin, (lastLogin) => lastLogin < ( now - 1.year ) ); // Select the first user that match userName and hashedPassword in the default store, with a challenge-response login auto loginUser = User.all. .filterUsing!(User.name, (name) => name == userName ) .filterUsing!(User.password, (pass) => hmac( pass, key ) == hashedPassword ) .onlyOne(); Gianni Pisetta
Jan 06
prev sibling parent reply Craig Dillabaugh <craig.dillabaugh gmail.com> writes:
On Thursday, 31 December 2015 at 17:14:55 UTC, Piotrek wrote:
 The goal of this post is to measure the craziness of an idea to 
 embed a database engine into the D language ;)

 I think about a database engine which would meet my three main 
 requirements:
   - integrated with D (ranges)
   - ACID
   - fast

 Since the days when I was working on financing data SW I become 
 allergic to SQL. I though that NoSQL databases would fill the 
 bill. Unfortunately they didn't. And I want to have an ability 
 to write a code like this without too much effort:

   struct Person
   {
    string name;
    string surname;
    ubyte age;
    Address address;
   }

  DataBase db = new DataBase("file.db");
  auto coll = db.collection!Person("NSA.Registry");
  auto visitationList = coll.filter!(p => p.name == "James");
  writeln (visitationList);

 And other things like updating and deleting from db. I think 
 you get my point.

 So I started a PoC project based on SQLite design:
 https://github.com/PiotrekDlang/AirLock/blob/master/docs/database/design.md#architecture

 The PoC code: 
 https://github.com/PiotrekDlang/AirLock/tree/master/src/database

 Can you please share your thoughts and experience on the topic? 
 Has anyone tried similar things?

 Piotrek
I've scanned this thread, but haven't seen if any 'decisions' have been more, or if it is just more of the usual back-and-forth with nothing being decided. However, I did have one (Ok two) questions. 1. Is there a GSOC project in here somewhere? 2. Who would want to mentor such a thing?
Feb 06
parent Craig Dillabaugh <craig.dillabaugh gmail.com> writes:
On Saturday, 6 February 2016 at 13:33:34 UTC, Craig Dillabaugh 
wrote:
clip
 I've scanned this thread, but haven't seen if any 'decisions' 
 have been more, or if it is just more of the usual
have been more => have been made
Feb 06