www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Abstract Database Interface

reply "BLM768" <blm768 gmail.com> writes:
I've recently been working with Ruby's ActiveRecord as part of my 
job, and I realized that D was powerful enough to make a similar 
abstraction layer. I've been playing with the idea for a little 
while, and I've put up some code at 
https://github.com/blm768/adbi. It isn't nearly as comprehensive 
as ActiveRecord, but it captures the basic idea of representing 
database structures as objects/structs.

Using it is something like this:

module main;

import std.stdio;

import adbi.model;
import adbi.sqlite3.database;

struct Test {
	mixin Model!"test";
	
	const(char)[] name;
	double num;
	mixin reference!("test2", "test2", Test2);
}

struct Test2 {
	mixin Model!"test2";
	int value;
}

int main(string[] args) {
	auto db = new Sqlite3Database("test.db");
	auto q = db.query("SELECT * FROM test;");
	
	Test.updateSchema(db);
	Test2.updateSchema(db);
	
	auto range = ModelRange!Test(q);
	
	foreach(value; range) {
		writeln(value);
	}
	
	auto q2 = db.query("SELECT * FROM test, test2 WHERE test2_id = 
test2.id");
	
	auto r2 = ModelRange!(Join!(Test, Test2))(q2);
	
	foreach(j; r2) {
		writeln(j);
	}
	
	return 0;
}

This code prints out every entry in the "test" table, then prints 
the results of a join on the "test" and "test2" tables. The calls 
to updateSchema() set up some static members of Test and Test2; 
after these calls, the library does not perform any operations 
with the column names, which should make retrieving a record 
noticeably faster than in a system like ActiveRecord. The 
downside is that these functions must be called every time the 
database schema changes in a way that affects column order, but 
that should happen very rarely, if ever, in a typical application.

The code is far from complete, but it's an interesting toy and 
might actually be useful for simple applications once some of the 
issues are ironed out.
Oct 27 2012
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-10-28 02:31, BLM768 wrote:
 I've recently been working with Ruby's ActiveRecord as part of my job,
 and I realized that D was powerful enough to make a similar abstraction
 layer. I've been playing with the idea for a little while, and I've put
 up some code at https://github.com/blm768/adbi. It isn't nearly as
 comprehensive as ActiveRecord, but it captures the basic idea of
 representing database structures as objects/structs.

 Using it is something like this:

 module main;

 import std.stdio;

 import adbi.model;
 import adbi.sqlite3.database;

 struct Test {
      mixin Model!"test";

      const(char)[] name;
      double num;
      mixin reference!("test2", "test2", Test2);
 }

 struct Test2 {
      mixin Model!"test2";
      int value;
 }

 int main(string[] args) {
      auto db = new Sqlite3Database("test.db");
      auto q = db.query("SELECT * FROM test;");

      Test.updateSchema(db);
      Test2.updateSchema(db);

      auto range = ModelRange!Test(q);

      foreach(value; range) {
          writeln(value);
      }

      auto q2 = db.query("SELECT * FROM test, test2 WHERE test2_id =
 test2.id");

      auto r2 = ModelRange!(Join!(Test, Test2))(q2);

      foreach(j; r2) {
          writeln(j);
      }

      return 0;
 }

 This code prints out every entry in the "test" table, then prints the
 results of a join on the "test" and "test2" tables. The calls to
 updateSchema() set up some static members of Test and Test2; after these
 calls, the library does not perform any operations with the column
 names, which should make retrieving a record noticeably faster than in a
 system like ActiveRecord. The downside is that these functions must be
 called every time the database schema changes in a way that affects
 column order, but that should happen very rarely, if ever, in a typical
 application.

 The code is far from complete, but it's an interesting toy and might
 actually be useful for simple applications once some of the issues are
 ironed out.
Looking at the API used in this example it would say that it's not very interesting and not very ActiveRecrod like. I think this looks more interesting and more like ActiveRecrod: class Person : Model { } void main () { auto p = new Person; p.name = "John Doe"; p.save(); p = Person.where!(x => x.name == "John Doe"); } But when you start to use associative it won't be as nice looking as ActiveRecord due to the not so nice mixin syntax. What we need is AST macros and user defined attributes/annotations. With that, associations could potentially look like this: class Foo : Model {} class Person : Model { hasMany Foo; } -- /Jacob Carlborg
Oct 28 2012
parent reply "BLM768" <blm768 gmail.com> writes:
 Looking at the API used in this example it would say that it's 
 not very interesting and not very ActiveRecrod like. I think 
 this looks more interesting and more like ActiveRecrod:

 class Person : Model
 {
 }

 void main ()
 {
     auto p = new Person;
     p.name = "John Doe";
     p.save();

     p = Person.where!(x => x.name == "John Doe");
 }

 But when you start to use associative it won't be as nice 
 looking as ActiveRecord due to the not so nice mixin syntax. 
 What we need is AST macros and user defined 
 attributes/annotations. With that, associations could 
 potentially look like this:

 class Foo : Model {}

 class Person : Model
 {
      hasMany Foo;
 }
It's definitely not ActiveRecord, but my goal is just to take some inspiration from it, not to duplicate it. I'm very concerned about efficiency, which is why I'm using structs, and I like hard-coding the fields into the structure so there's some documentation of what the record is supposed to hold and so the compiler can optimize it more heavily. It will probably be a little less pretty, but it'll work, and that's what really matters. At some point, I might implement an interface to generate SQL queries with function calls, but for now, just manually writing the queries really isn't hard, and it provides a significant speed boost for systems like Sqlite that compile queries down to bytecode because it's easier to reuse the query object.
Oct 29 2012
parent reply Jacob Carlborg <doob me.com> writes:
 It's definitely not ActiveRecord, but my goal is just to take some
 inspiration from it, not to duplicate it. I'm very concerned about
 efficiency, which is why I'm using structs, and I like hard-coding the
 fields into the structure so there's some documentation of what the
 record is supposed to hold and so the compiler can optimize it more
 heavily. It will probably be a little less pretty, but it'll work, and
 that's what really matters. At some point, I might implement an
 interface to generate SQL queries with function calls, but for now, just
 manually writing the queries really isn't hard, and it provides a
 significant speed boost for systems like Sqlite that compile queries
 down to bytecode because it's easier to reuse the query object.
My point was just that you removed the key features and soul of ActiveRecord. Without these features it's just like any other ORM library. -- /Jacob Carlborg
Oct 29 2012
parent reply "BLM768" <blm768 gmail.com> writes:
 My point was just that you removed the key features and soul of 
 ActiveRecord. Without these features it's just like any other 
 ORM library.
That's a good point. I haven't had any experience with other ORM libraries, so ActiveRecord was the closest thing that came to mind. I definitely do want to eventually capture some of ActiveRecord's features, but probably not all of them. I feel like the solution should be implemented in a way that fits well with a statically typed language, so I'll definitely have to drop some of the features. It won't be quite as nice to use, but it will be simpler in dome ways, which is one of my primary goals as a developer. Tools like ActiveRecord are more fun to use, but thinking of all the hash table lookups makes me cringe. :) If and when the library matures, though, I might think about adding some more ActiveRecord-like features if enough people miss them.
Oct 29 2012
parent reply Jacob Carlborg <doob me.com> writes:
On 2012-10-29 15:42, BLM768 wrote:

 That's a good point. I haven't had any experience with other ORM
 libraries, so ActiveRecord was the closest thing that came to mind.

 I definitely do want to eventually capture some of ActiveRecord's
 features, but probably not all of them. I feel like the solution should
 be implemented in a way that fits well with a statically typed language,
 so I'll definitely have to drop some of the features. It won't be quite
 as nice to use, but it will be simpler in dome ways, which is one of my
 primary goals as a developer. Tools like ActiveRecord are more fun to
 use, but thinking of all the hash table lookups makes me cringe. :)

 If and when the library matures, though, I might think about adding some
 more ActiveRecord-like features if enough people miss them.
You can have a look at DataMapper. That's also for Ruby but it's not specific for SQL, if I recall correctly. Have a look at some ORM library written in Scala, I would guess they can be quite innovative and it's statically typed. http://squeryl.org/index.html http://datamapper.org/ -- /Jacob Carlborg
Oct 29 2012
parent reply "BLM768" <blm768 gmail.com> writes:
 You can have a look at DataMapper. That's also for Ruby but 
 it's not specific for SQL, if I recall correctly. Have a look 
 at some ORM library written in Scala, I would guess they can be 
 quite innovative and it's statically typed.

 http://squeryl.org/index.html
 http://datamapper.org/
Those libraries definitely look interesting. I should probably consider some type of NoSQL database support... Thanks for the links!
Oct 29 2012
parent reply Jacob Carlborg <doob me.com> writes:
On 2012-10-29 18:43, BLM768 wrote:
 You can have a look at DataMapper. That's also for Ruby but it's not
 specific for SQL, if I recall correctly. Have a look at some ORM
 library written in Scala, I would guess they can be quite innovative
 and it's statically typed.

 http://squeryl.org/index.html
 http://datamapper.org/
Those libraries definitely look interesting. I should probably consider some type of NoSQL database support...
If I recall correctly, Squeryl use Scala AST macros to support a query syntax, that in D would look, as below: class Person : Model { } void main () { auto p = new Person; p.name = "John Doe"; p.save(); p = Person.where!(x => x.name == "John Doe"); } -- /Jacob Carlborg
Oct 29 2012
parent reply "BLM768" <blm768 gmail.com> writes:
 If I recall correctly, Squeryl use Scala AST macros to support 
 a query syntax, that in D would look, as below:

 class Person : Model { }

 void main ()
 {
     auto p = new Person;
     p.name = "John Doe";
     p.save();

     p = Person.where!(x => x.name == "John Doe");
 }
If you make x some fancy wrapper type containing more fancy wrapper types with overloaded equality operators that return some sort of Expression class instead of a boolean, you might actually be able to get this to work with only D's current features. However, that would kind of destroy the hope of efficiency. :) What might be nice is a database written in D that completely eschews SQL in favor of a native API. I might have to play with that eventually, but I'll probably give it a while because it would be a huge project, and, like most people, I'm under time constraints. :)
Oct 29 2012
parent reply Jacob Carlborg <doob me.com> writes:
On 2012-10-30 04:22, BLM768 wrote:

 If you make x some fancy wrapper type containing more fancy wrapper
 types with overloaded equality operators that return some sort of
 Expression class instead of a boolean, you might actually be able to get
 this to work with only D's current features. However, that would kind of
 destroy the hope of efficiency. :)
It can probably all be handled at compile time. The problem with this that you cannot overload the following operators: &&, ||, != and probably some other useful operators.
 What might be nice is a database written in D that completely eschews
 SQL in favor of a native API. I might have to play with that eventually,
 but I'll probably give it a while because it would be a huge project,
 and, like most people, I'm under time constraints. :)
Yeah, I know. -- /Jacob Carlborg
Oct 30 2012
parent reply Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Tue, Oct 30, 2012 at 9:15 AM, Jacob Carlborg <doob me.com> wrote:
 On 2012-10-30 04:22, BLM768 wrote:

 If you make x some fancy wrapper type containing more fancy wrapper
 types with overloaded equality operators that return some sort of
 Expression class instead of a boolean, you might actually be able to get
 this to work with only D's current features. However, that would kind of
 destroy the hope of efficiency. :)
It can probably all be handled at compile time. The problem with this that you cannot overload the following operators: &&, ||, != and probably some other useful operators.
&& and || can be replaced by & and |, so there is a workaround. I feel much more limited by != and, even more problematic, !. Maybe unary - could be used in lieu of !.
Oct 30 2012
next sibling parent reply "Kapps" <opantm2+spam gmail.com> writes:
On Tuesday, 30 October 2012 at 10:01:06 UTC, Philippe Sigaud 
wrote:
 On Tue, Oct 30, 2012 at 9:15 AM, Jacob Carlborg <doob me.com> 
 wrote:
 On 2012-10-30 04:22, BLM768 wrote:

 If you make x some fancy wrapper type containing more fancy 
 wrapper
 types with overloaded equality operators that return some 
 sort of
 Expression class instead of a boolean, you might actually be 
 able to get
 this to work with only D's current features. However, that 
 would kind of
 destroy the hope of efficiency. :)
It can probably all be handled at compile time. The problem with this that you cannot overload the following operators: &&, ||, != and probably some other useful operators.
&& and || can be replaced by & and |, so there is a workaround. I feel much more limited by != and, even more problematic, !. Maybe unary - could be used in lieu of !.
There was a pull request for __traits(codeof, func) that would return the code for a symbol including lambda methods. It would probably be easier to have something like that for getting the LINQ does it) than using sketchy hacks that go against natural language feel. Though it wouldn't be particularly easy to get that in to the compiler apparently due to AST rewriting issues. https://github.com/D-Programming-Language/dmd/pull/953
Oct 30 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-10-30 13:04, Kapps wrote:

 There was a pull request for __traits(codeof, func) that would return
 the code for a symbol including lambda methods. It would probably be
 easier to have something like that for getting the AST and then using

 sketchy hacks that go against natural language feel. Though it wouldn't
 be particularly easy to get that in to the compiler apparently due to
 AST rewriting issues.
How would that work in this case, the code need to compile? I mean, even if you can get the syntax of a function, process it correctly and generate SQL from it, the function still need to compile. -- /Jacob Carlborg
Oct 30 2012
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-10-30 10:59, Philippe Sigaud wrote:

 && and || can be replaced by & and |, so there is a workaround.
 I feel much more limited by != and, even more problematic, !.  Maybe
 unary - could be used in lieu of !.
How does that work with operator precedence? There's a plugin for ActiveRecord, called Squeel, that allows you to do something like this: Person.where do |q| (q.name == "asd") & (q.address == "foo") end But because of the operator precedence in Ruby you need to wrap every comparison in parentheses, not very pretty. -- /Jacob Carlborg
Oct 30 2012
parent reply Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Tue, Oct 30, 2012 at 3:44 PM, Jacob Carlborg <doob me.com> wrote:

 How does that work with operator precedence?
(...)
 But because of the operator precedence in Ruby you need to wrap every
 comparison in parentheses, not very pretty.
I think the problem would the same here. Of course, to know D operator precedence, you have to dig into the grammar, since there is no handy table to give you that info :)
Oct 30 2012
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 10/30/2012 04:47 PM, Philippe Sigaud wrote:
 On Tue, Oct 30, 2012 at 3:44 PM, Jacob Carlborg <doob me.com> wrote:

 How does that work with operator precedence?
(...)
 But because of the operator precedence in Ruby you need to wrap every
 comparison in parentheses, not very pretty.
I think the problem would the same here. Of course, to know D operator precedence, you have to dig into the grammar, since there is no handy table to give you that info :)
From higher to lower, where relational ops are unordered with respect to bitwise ops (this is the reason comparisons would have to be wrapped in parentheses in D as well): ! => (not a real operator, occurs twice this is binding power to the left) . ++ -- ( [ ^^ (right-associative) & ++ -- * - + ! ~ (prefix) * / % + - ~ << >> >>> == != > < >= <= !> !< !>= !<= <> !<> <>= !<>= in !in is !is & ^ | && || ? (right-associative) /= &= |= -= += <<= >>= >>>= = *= %= ^= ^^= ~= (right-associative) => (not a real operator, occurs twice, this is binding power to the right) , .. (not a real operator)
Oct 30 2012
parent reply Philippe Sigaud <philippe.sigaud gmail.com> writes:
Timon:

 From higher to lower, where relational ops are unordered with respect to
 bitwise ops (this is the reason comparisons would have to be wrapped in
 parentheses in D as well):

 !
 => (not a real operator, occurs twice this is binding power to the left)
 . ++ -- ( [
 ^^ (right-associative)
 & ++ -- * - + ! ~ (prefix)
 * / %
 + - ~
 << >> >>>
 == != > < >= <= !> !< !>= !<= <> !<> <>= !<>= in !in is !is
 &
 ^
 |
 &&
 ||
 ? (right-associative)
 /= &= |= -= += <<= >>= >>>= = *= %= ^= ^^= ~= (right-associative)
 => (not a real operator, occurs twice, this is binding power to the right)
 ,
 .. (not a real operator)
Thanks a lot Timon! => is indeed a strange beast. Any possibility to put that on a wiki page or the dlang.org on operator overloading?
Oct 31 2012
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 10/31/2012 05:54 PM, Philippe Sigaud wrote:
 Timon:

  From higher to lower, where relational ops are unordered with respect to
 bitwise ops (this is the reason comparisons would have to be wrapped in
 parentheses in D as well):

 !
 => (not a real operator, occurs twice this is binding power to the left)
 . ++ -- ( [
 ^^ (right-associative)
 & ++ -- * - + ! ~ (prefix)
 * / %
 + - ~
 << >> >>>
 == != > < >= <= !> !< !>= !<= <> !<> <>= !<>= in !in is !is
 &
 ^
 |
 &&
 ||
 ? (right-associative)
 /= &= |= -= += <<= >>= >>>= = *= %= ^= ^^= ~= (right-associative)
 => (not a real operator, occurs twice, this is binding power to the right)
 ,
 .. (not a real operator)
Thanks a lot Timon! => is indeed a strange beast. Any possibility to put that on a wiki page or the dlang.org on operator overloading?
http://wiki.dlang.org/DWiki:Operator_precedence
Dec 23 2012
parent reply Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Sun, Dec 23, 2012 at 7:37 PM, Timon Gehr <timon.gehr gmx.ch> wrote:

 On 10/31/2012 05:54 PM, Philippe Sigaud wrote:

 => is indeed a strange beast.
Any possibility to put that on a wiki page or the dlang.org on operator overloading?
http://wiki.dlang.org/DWiki:**Operator_precedence<http://wiki.dlang.org/DWiki:Operator_precedence>
Seems good. I remember levels 6-8 from the online grammar. I guess it feels natural when we use them. The wiki has a much better appearance, I'll try to participate and fill some parts.
Dec 24 2012
parent "BLM768" <blm768 gmail.com> writes:
I was kind of shocked to find this thread at the top of the list 
again; I thought it had died. :) This is a very interesting time 
for the thread to revive because I was just thinking about the 
project this morning...

Now that my interest has revived a bit, I'm trying to figure out 
how to get SQLite working on Windows; the only real options I've 
seen so far are to compile SQLite with DMC or to use GDC instead 
of DMD. Creating a .lib from sqlite3.dll looks like too much of a 
pain.

I'm also thinking about adding a PosgreSQL driver; I'll probably 
use code from SQLd for that.
Dec 24 2012
prev sibling next sibling parent Rory McGuire <rjmcguire gmail.com> writes:
I like, nice work.
On 28 Oct 2012 02:35, "BLM768" <blm768 gmail.com> wrote:

 I've recently been working with Ruby's ActiveRecord as part of my job, and
 I realized that D was powerful enough to make a similar abstraction layer.
 I've been playing with the idea for a little while, and I've put up some
 code at https://github.com/blm768/adbi**. It isn't nearly as
 comprehensive as ActiveRecord, but it captures the basic idea of
 representing database structures as objects/structs.

 Using it is something like this:

 module main;

 import std.stdio;

 import adbi.model;
 import adbi.sqlite3.database;

 struct Test {
         mixin Model!"test";

         const(char)[] name;
         double num;
         mixin reference!("test2", "test2", Test2);
 }

 struct Test2 {
         mixin Model!"test2";
         int value;
 }

 int main(string[] args) {
         auto db = new Sqlite3Database("test.db");
         auto q = db.query("SELECT * FROM test;");

         Test.updateSchema(db);
         Test2.updateSchema(db);

         auto range = ModelRange!Test(q);

         foreach(value; range) {
                 writeln(value);
         }

         auto q2 = db.query("SELECT * FROM test, test2 WHERE test2_id =
 test2.id");

         auto r2 = ModelRange!(Join!(Test, Test2))(q2);

         foreach(j; r2) {
                 writeln(j);
         }

         return 0;
 }

 This code prints out every entry in the "test" table, then prints the
 results of a join on the "test" and "test2" tables. The calls to
 updateSchema() set up some static members of Test and Test2; after these
 calls, the library does not perform any operations with the column names,
 which should make retrieving a record noticeably faster than in a system
 like ActiveRecord. The downside is that these functions must be called
 every time the database schema changes in a way that affects column order,
 but that should happen very rarely, if ever, in a typical application.

 The code is far from complete, but it's an interesting toy and might
 actually be useful for simple applications once some of the issues are
 ironed out.
Oct 28 2012
prev sibling parent reply "Robik" <szadows gmail.com> writes:
On Sunday, 28 October 2012 at 00:31:49 UTC, BLM768 wrote:
 I've recently been working with Ruby's ActiveRecord as part of 
 my job, and I realized that D was powerful enough to make a 
 similar abstraction layer. I've been playing with the idea for 
 a little while, and I've put up some code at 
 https://github.com/blm768/adbi. It isn't nearly as 
 comprehensive as ActiveRecord, but it captures the basic idea 
 of representing database structures as objects/structs.

 Using it is something like this:

 module main;

 import std.stdio;

 import adbi.model;
 import adbi.sqlite3.database;

 struct Test {
 	mixin Model!"test";
 	
 	const(char)[] name;
 	double num;
 	mixin reference!("test2", "test2", Test2);
 }

 struct Test2 {
 	mixin Model!"test2";
 	int value;
 }

 int main(string[] args) {
 	auto db = new Sqlite3Database("test.db");
 	auto q = db.query("SELECT * FROM test;");
 	
 	Test.updateSchema(db);
 	Test2.updateSchema(db);
 	
 	auto range = ModelRange!Test(q);
 	
 	foreach(value; range) {
 		writeln(value);
 	}
 	
 	auto q2 = db.query("SELECT * FROM test, test2 WHERE test2_id = 
 test2.id");
 	
 	auto r2 = ModelRange!(Join!(Test, Test2))(q2);
 	
 	foreach(j; r2) {
 		writeln(j);
 	}
 	
 	return 0;
 }

 This code prints out every entry in the "test" table, then 
 prints the results of a join on the "test" and "test2" tables. 
 The calls to updateSchema() set up some static members of Test 
 and Test2; after these calls, the library does not perform any 
 operations with the column names, which should make retrieving 
 a record noticeably faster than in a system like ActiveRecord. 
 The downside is that these functions must be called every time 
 the database schema changes in a way that affects column order, 
 but that should happen very rarely, if ever, in a typical 
 application.

 The code is far from complete, but it's an interesting toy and 
 might actually be useful for simple applications once some of 
 the issues are ironed out.
I am working on similar project, named SQLd[1]. If you are interested, we can join forces and work togheder :) IRC Nick: Robik [1]: http://github.com/robik/SQLd
Oct 28 2012
parent reply "BLM768" <blm768 gmail.com> writes:
 I am working on similar project, named SQLd[1]. If you are 
 interested, we can join forces and work togheder :)

 IRC Nick: Robik
 [1]: http://github.com/robik/SQLd
That might be a good idea. I haven't done much for supporting different databases, so getting more backend support would be quite nice. What might work well is for me to just refactor my code to sit on top of your existing database classes. So far, I've really just been playing around, but if people show enough interest, I'd like to play with the idea a while longer. :)
Oct 29 2012
parent Marco Leise <Marco.Leise gmx.de> writes:
Am Mon, 29 Oct 2012 08:50:19 +0100
schrieb "BLM768" <blm768 gmail.com>:

 
 I am working on similar project, named SQLd[1]. If you are 
 interested, we can join forces and work togheder :)

 IRC Nick: Robik
 [1]: http://github.com/robik/SQLd
That might be a good idea. I haven't done much for supporting different databases, so getting more backend support would be quite nice. What might work well is for me to just refactor my code to sit on top of your existing database classes. So far, I've really just been playing around, but if people show enough interest, I'd like to play with the idea a while longer. :)
I would use such an ORM library, too. It is definitely worth the effort. Currently I use something that just helps me a bit with fetching and updating entries in tables, but goes in the same direction of using only structs for table rows. Here are some database bindings from wiki4d that others have written: http://mleise.dnsd.info/#DatabaseDrivers Some are for D1 though. But now that you are team you may want to look at those bindings for some inspiration, so you don't get locked in on SQLite3 too much :p -- Marco
Nov 06 2012