www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - [RFC] - mysql-native rewrite

reply "simendsjo" <simendsjo gmail.com> writes:
I've been working on a more or less complete rewrite of the 
mysql-native module.

The main focus has been on making the source better decoupled and 
more maintainable.
In the process, I've changed the API too...

The existing production code can be found here: 
https://github.com/rejectedsoftware/mysql-native

New code: 
http://simendsjo.me/files/simendsjo/mysqln-rewrite/mysqln-rewrite.7z
Documentation: 
http://simendsjo.me/files/simendsjo/mysqln-rewrite/docs/

The files:
* adhoc.d - Execute simple queries. This is meant to be the new 
high-level public API
* commands.d - Lower level public API
* result.d - Common interface for both Text and Binary MySQL 
results
* connection.d - Phobos/Vibe.d connection
* db.d - Vibe.d connection pool

The rest are just internal files.

Todo:
* Implement all MySQL types
* Prepared statement release/purge
* Lazily fetch data
* Fix bugs in prepared statement binding
* Implement SEND_LONG_DATA
* More integrationtests and unittests

Most of the above would be pretty simple to implement.

I'm very uncertain about several aspects of my design:
* Class vs. struct
* Returned values from MySQL - e.g. should SELECT TRUE return 
long as it does in MySQL, or should we interpret it as bool
* Probably a lot I don't remember right now :)

Any comments would be very helpful.
..Or in D terms: DESTROY!
Sep 28 2013
next sibling parent "Gary Willoughby" <dev nomad.so> writes:
On Saturday, 28 September 2013 at 16:39:27 UTC, simendsjo wrote:

 I've been working on a more or less complete rewrite of the 
 mysql-native module.

 The main focus has been on making the source better decoupled 
 and more maintainable.
 In the process, I've changed the API too...

 The existing production code can be found here: 
 https://github.com/rejectedsoftware/mysql-native

 New code: 
 http://simendsjo.me/files/simendsjo/mysqln-rewrite/mysqln-rewrite.7z
 Documentation: 
 http://simendsjo.me/files/simendsjo/mysqln-rewrite/docs/

 The files:
 * adhoc.d - Execute simple queries. This is meant to be the new 
 high-level public API
 * commands.d - Lower level public API
 * result.d - Common interface for both Text and Binary MySQL 
 results
 * connection.d - Phobos/Vibe.d connection
 * db.d - Vibe.d connection pool

 The rest are just internal files.

 Todo:
 * Implement all MySQL types
 * Prepared statement release/purge
 * Lazily fetch data
 * Fix bugs in prepared statement binding
 * Implement SEND_LONG_DATA
 * More integrationtests and unittests

 Most of the above would be pretty simple to implement.

 I'm very uncertain about several aspects of my design:
 * Class vs. struct
 * Returned values from MySQL - e.g. should SELECT TRUE return 
 long as it does in MySQL, or should we interpret it as bool
 * Probably a lot I don't remember right now :)

 Any comments would be very helpful.
 ..Or in D terms: DESTROY!


This sounds great! I use this library a lot so i can probably 
give you some feedback when i have more time. I'm about to go out 
now but i'll try to take a look tomorrow.
Sep 28 2013
prev sibling next sibling parent reply "Gary Willoughby" <dev nomad.so> writes:
On Saturday, 28 September 2013 at 16:39:27 UTC, simendsjo wrote:

 I'm very uncertain about several aspects of my design:
 * Class vs. struct
 * Returned values from MySQL - e.g. should SELECT TRUE return 
 long as it does in MySQL, or should we interpret it as bool
 * Probably a lot I don't remember right now :)

 Any comments would be very helpful.
 ..Or in D terms: DESTROY!


Ok, i've taken a quick look and like where this is going. I'll 
try and give you a little guide to what i would do regarding the 
above points.


 * Class vs. struct


I tend to only use structs where the type is purely a data type. 
Something like a record or a type that can be manifested in 
different ways (such as an IP address). If i need to model a real 
world object like an engine or book, etc., i use a class.


 * Returned values from MySQL - e.g. should SELECT TRUE return 
 long as it does in MySQL, or should we interpret it as bool


I would return what mysql yields. This would make sure this 
framework is not doing to much. Converting ints to bools would be 
the next layer's job (DBAL, ORM, etc.).

I have two more suggestions that i think is critical to a project 
such as this and that's documentation and unit tests.

Please from the start thoroughly annotate everything with ddoc to 
generate nice html docs later. I know this can be a pain when 
designing and developing an API but is absolutely necessary. If 
you leave documentation until the end it never gets done!

Write unit tests for everything. I've found that if you find it 
hard to write a unit test for any 'unit', etc then it's too 
tightly coupled and probably doing too much. Practise good SOLID 
design principles and unit tests should be easy to write. Not 
only that but unit tests provide developers with a good 
understanding of the public interface.

http://en.wikipedia.org/wiki/SOLID_(object-oriented_design)
Sep 29 2013
parent reply "simendsjo" <simendsjo gmail.com> writes:
On Sunday, 29 September 2013 at 15:26:19 UTC, Gary Willoughby 
wrote:

 On Saturday, 28 September 2013 at 16:39:27 UTC, simendsjo wrote:

 I'm very uncertain about several aspects of my design:
 * Class vs. struct
 * Returned values from MySQL - e.g. should SELECT TRUE return 
 long as it does in MySQL, or should we interpret it as bool
 * Probably a lot I don't remember right now :)

 Any comments would be very helpful.
 ..Or in D terms: DESTROY!


 Ok, i've taken a quick look and like where this is going. I'll 
 try and give you a little guide to what i would do regarding 
 the above points.


 * Class vs. struct


 I tend to only use structs where the type is purely a data 
 type. Something like a record or a type that can be manifested 
 in different ways (such as an IP address). If i need to model a 
 real world object like an engine or book, etc., i use a class.


I don't think it's that simple in this case. When I implement 
lazy fetching, both methods have it's advantages and 
disadvantages.
MySQL doesn't allow multiplexing on a connection. This means a 
command must complete before issuing another. If reading rows 
lazily and then issuing a new command, the choice is to either 
disallow the new command, or invalidate the previous. The simple 
way would be to just start a new command and invalidate the 
previous, but is this the best way? If we choose to disallow new 
commands, that means the user have to explicitly close a lazy 
command. If using classes, we can safely have several instances 
for a command (is this neccessary?), but then the destructor 
wont't be a safe bet, and the user have to call close.
If implemented as a struct, we have to disallow copying.

So... I really don't know what the best design would be.


 * Returned values from MySQL - e.g. should SELECT TRUE return 
 long as it does in MySQL, or should we interpret it as bool


 I would return what mysql yields. This would make sure this 
 framework is not doing to much. Converting ints to bools would 
 be the next layer's job (DBAL, ORM, etc.).


The problem is that MySQL is inconsistent here, and it depends on 
if it's a field or a constant. SELECT TRUE is not the same as 
SELECT boolean_field.
Also, it seems every constant integer is returned as LONGLONG. 
But.. The fields include a length property..


 I have two more suggestions that i think is critical to a 
 project such as this and that's documentation and unit tests.

 Please from the start thoroughly annotate everything with ddoc 
 to generate nice html docs later. I know this can be a pain 
 when designing and developing an API but is absolutely 
 necessary. If you leave documentation until the end it never 
 gets done!


Yeah, the documentation isn't very good :/

Thought I'd get feedback on the API before spending a lot of time 
documenting it.


 Write unit tests for everything. I've found that if you find it 
 hard to write a unit test for any 'unit', etc then it's too 
 tightly coupled and probably doing too much. Practise good 
 SOLID design principles and unit tests should be easy to write. 
 Not only that but unit tests provide developers with a good 
 understanding of the public interface.

 http://en.wikipedia.org/wiki/SOLID_(object-oriented_design)


Most of the code is easy to unittest, but for now I've just 
relied on integration tests against a database. I'll improve on 
the situation.
Sep 29 2013
next sibling parent "Gary Willoughby" <dev nomad.so> writes:
On Sunday, 29 September 2013 at 15:42:10 UTC, simendsjo wrote:

 I don't think it's that simple in this case. When I implement 
 lazy fetching, both methods have it's advantages and 
 disadvantages.
 MySQL doesn't allow multiplexing on a connection. This means a 
 command must complete before issuing another. If reading rows 
 lazily and then issuing a new command, the choice is to either 
 disallow the new command, or invalidate the previous. The 
 simple way would be to just start a new command and invalidate 
 the previous, but is this the best way? If we choose to 
 disallow new commands, that means the user have to explicitly 
 close a lazy command. If using classes, we can safely have 
 several instances for a command (is this neccessary?), but then 
 the destructor wont't be a safe bet, and the user have to call 
 close.
 If implemented as a struct, we have to disallow copying.

 So... I really don't know what the best design would be.


I reckon go with Command being a class and invalidate previous 
lazy loading on new commands being issued, you are after all 
issuing a *new* command. If multiplexing is needed a new 
connection can be made.
Sep 29 2013
prev sibling parent reply "Kagamin" <spam here.lot> writes:
On Sunday, 29 September 2013 at 15:42:10 UTC, simendsjo wrote:

 the previous, but is this the best way? If we choose to 
 disallow new commands, that means the user have to explicitly 
 close a lazy command.


If a command isn't closed as soon as possible, it sounds like a 
resource leak. If you silently cancel previous query, it can hide 
logical bug.


 If using classes, we can safely have several instances for a 
 command (is this neccessary?)


If you execute the same query repeatedly, there's no need to 
build the same command every time.


 The problem is that MySQL is inconsistent here, and it depends 
 on if it's a field or a constant. SELECT TRUE is not the same 
 as SELECT boolean_field.
 Also, it seems every constant integer is returned as LONGLONG. 
 But.. The fields include a length property..


Is there some hint it's true and not something else?
Sep 30 2013
parent reply "simendsjo" <simendsjo gmail.com> writes:
On Monday, 30 September 2013 at 10:34:25 UTC, Kagamin wrote:

 On Sunday, 29 September 2013 at 15:42:10 UTC, simendsjo wrote:

 the previous, but is this the best way? If we choose to 
 disallow new commands, that means the user have to explicitly 
 close a lazy command.


 If a command isn't closed as soon as possible, it sounds like a 
 resource leak. If you silently cancel previous query, it can 
 hide logical bug.


Yeah. We need to choose:
1) Starting a new command while another is in flight is an error
    You need to close explicitly if the command isn't finished
2) If another command has been started, it's er error to continue 
iteration of a previous command.

I'm in favor of 1).


 If using classes, we can safely have several instances for a 
 command (is this neccessary?)


 If you execute the same query repeatedly, there's no need to 
 build the same command every time.


That's what prepared statements are for. If you don't even have 
parameters, it's just 9 bytes (4 of these bytes is an 
undocumented field called iteration_count..).


 The problem is that MySQL is inconsistent here, and it depends 
 on if it's a field or a constant. SELECT TRUE is not the same 
 as SELECT boolean_field.
 Also, it seems every constant integer is returned as LONGLONG. 
 But.. The fields include a length property..


 Is there some hint it's true and not something else?


Yeah. BOOL is an alias for TINYINT(1). I think it's fair to 
always assume TINYINT(1) is bool. For other types I'm not quite 
sure though.. "SELECT 1" is a LONGLONG, but in D a literal is int 
unless specified otherwise. So there is a mismatch here.
Sep 30 2013
next sibling parent reply "Kagamin" <spam here.lot> writes:
On Monday, 30 September 2013 at 11:20:23 UTC, simendsjo wrote:

 Yeah. We need to choose:
 1) Starting a new command while another is in flight is an error
    You need to close explicitly if the command isn't finished
 2) If another command has been started, it's er error to 
 continue iteration of a previous command.

 I'm in favor of 1).


That's what .net does. Most of the time it indicates a resource 
leak.


 Yeah. BOOL is an alias for TINYINT(1). I think it's fair to 
 always assume TINYINT(1) is bool. For other types I'm not quite 
 sure though.. "SELECT 1" is a LONGLONG, but in D a literal is 
 int unless specified otherwise. So there is a mismatch here.


int64 is the right type for a database integer, int32 is too 
small. After all, it's a database.
Sep 30 2013
next sibling parent reply "simendsjo" <simendsjo gmail.com> writes:
On Monday, 30 September 2013 at 11:58:48 UTC, Kagamin wrote:

 On Monday, 30 September 2013 at 11:20:23 UTC, simendsjo wrote:

 Yeah. We need to choose:
 1) Starting a new command while another is in flight is an 
 error
   You need to close explicitly if the command isn't finished
 2) If another command has been started, it's er error to 
 continue iteration of a previous command.

 I'm in favor of 1).


 That's what .net does. Most of the time it indicates a resource 
 leak.


Ok. I'll add the same semantics.


 Yeah. BOOL is an alias for TINYINT(1). I think it's fair to 
 always assume TINYINT(1) is bool. For other types I'm not 
 quite sure though.. "SELECT 1" is a LONGLONG, but in D a 
 literal is int unless specified otherwise. So there is a 
 mismatch here.


 int64 is the right type for a database integer, int32 is too 
 small. After all, it's a database.


I'll just add a hack for TINYINT(1) -> bool and keep everything 
else MySQL specific.

Have to admit that int32 has been plenty for me in the past. 
Never worked on databases with > 200M rows.
Sep 30 2013
next sibling parent "Kagamin" <spam here.lot> writes:
It's not only size. A popular use case is a log: you write lots 
of entries and delete old ones, total size doesn't grow, but the 
identifiers...
Sep 30 2013
prev sibling parent reply "Gary Willoughby" <dev nomad.so> writes:
On Monday, 30 September 2013 at 12:05:49 UTC, simendsjo wrote:

 I'll just add a hack for TINYINT(1) -> bool and keep everything 
 else MySQL specific.


Please don't. The db library should only return what mysql 
returns. Let the next layer handle casting types.


 Have to admit that int32 has been plenty for me in the past. 
 Never worked on databases with > 200M rows.


I routinely use db's with enormous datasets 500M+ and use BIGINTS 
a lot. If i select a BIGINT column i'd expect a long back.
Sep 30 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-09-30 15:07, Gary Willoughby wrote:


 Please don't. The db library should only return what mysql returns. Let
 the next layer handle casting types.


Agree.

-- 
/Jacob Carlborg
Sep 30 2013
parent "simendsjo" <simendsjo gmail.com> writes:
On Monday, 30 September 2013 at 13:33:09 UTC, Jacob Carlborg 
wrote:

 On 2013-09-30 15:07, Gary Willoughby wrote:


 Please don't. The db library should only return what mysql 
 returns. Let
 the next layer handle casting types.


 Agree.


Seems everyone agrees on this. In that case, I won't do any kind 
of processing of the types.
Sep 30 2013
prev sibling parent "Gary Willoughby" <dev nomad.so> writes:
On Monday, 30 September 2013 at 11:58:48 UTC, Kagamin wrote:

 int64 is the right type for a database integer, int32 is too 
 small. After all, it's a database.


All mysql types should be handled by using the correct type in D. 
e.g. if i select an INT i want a D int. If i select a BINGINT i 
want a D long, etc.
Sep 30 2013
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-09-30 13:20, simendsjo wrote:


 Yeah. We need to choose:
 1) Starting a new command while another is in flight is an error
     You need to close explicitly if the command isn't finished
 2) If another command has been started, it's er error to continue
 iteration of a previous command.

 I'm in favor of 1).


I would go with one as well.


 Yeah. BOOL is an alias for TINYINT(1). I think it's fair to always
 assume TINYINT(1) is bool. For other types I'm not quite sure though..
 "SELECT 1" is a LONGLONG, but in D a literal is int unless specified
 otherwise. So there is a mismatch here.


Use the type in D which has the closets match and have the same length. 
Type conversions can be made later in a higher level API.

-- 
/Jacob Carlborg
Sep 30 2013
prev sibling next sibling parent reply "Chris" <wendlec tcd.ie> writes:
On Saturday, 28 September 2013 at 16:39:27 UTC, simendsjo wrote:

 I've been working on a more or less complete rewrite of the 
 mysql-native module.

 The main focus has been on making the source better decoupled 
 and more maintainable.
 In the process, I've changed the API too...

 The existing production code can be found here: 
 https://github.com/rejectedsoftware/mysql-native

 New code: 
 http://simendsjo.me/files/simendsjo/mysqln-rewrite/mysqln-rewrite.7z
 Documentation: 
 http://simendsjo.me/files/simendsjo/mysqln-rewrite/docs/

 The files:
 * adhoc.d - Execute simple queries. This is meant to be the new 
 high-level public API
 * commands.d - Lower level public API
 * result.d - Common interface for both Text and Binary MySQL 
 results
 * connection.d - Phobos/Vibe.d connection
 * db.d - Vibe.d connection pool

 The rest are just internal files.

 Todo:
 * Implement all MySQL types
 * Prepared statement release/purge
 * Lazily fetch data
 * Fix bugs in prepared statement binding
 * Implement SEND_LONG_DATA
 * More integrationtests and unittests

 Most of the above would be pretty simple to implement.

 I'm very uncertain about several aspects of my design:
 * Class vs. struct
 * Returned values from MySQL - e.g. should SELECT TRUE return 
 long as it does in MySQL, or should we interpret it as bool
 * Probably a lot I don't remember right now :)

 Any comments would be very helpful.
 ..Or in D terms: DESTROY!


It would be nice, if someone with experience wrote a short 
article / tutorial about when to use structs and when to use 
classes, where the pitfalls are etc. It is not always easy to 
decide and a rough guide would be great. As of now I'm using 
structs and ranges only for reformatting input. I'm not sure, 
however, if it is a good idea to go beyond this (cf. different 
reference semantics, polymorphism etc.)
Sep 30 2013
parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 30 September 2013 at 14:00:10 UTC, Chris wrote:

 It would be nice, if someone with experience wrote a short 
 article / tutorial about when to use structs and when to use 
 classes, where the pitfalls are etc.


I don't think there is a common consensus here. Don't even think 
it is possible. Doom of proper multi-paradigm language :)
Sep 30 2013
next sibling parent "Chris" <wendlec tcd.ie> writes:
On Monday, 30 September 2013 at 14:08:12 UTC, Dicebot wrote:

 On Monday, 30 September 2013 at 14:00:10 UTC, Chris wrote:

 It would be nice, if someone with experience wrote a short 
 article / tutorial about when to use structs and when to use 
 classes, where the pitfalls are etc.


 I don't think there is a common consensus here. Don't even 
 think it is possible. Doom of proper multi-paradigm language :)


Well, if people shared their experience with ranges / structs vs. 
classes, it would be a good reference point for other 
programmers. There might not be any real rules, but rough 
guidelines would help, e.g. "If you want to achieve X, structs 
are better, because blah blah. But if you want to do Y, there 
might be a problem with structs. Better use classes." Maybe 
structs are better suited for parallel processing and multi 
threading (value vs reference), or maybe not? Just a collection 
of little hints would make life simpler. I'm sure someone has 
shot him/herself seriously in the foot using either structs or 
classes.

I use structs mainly for a. formatting and reformatting (I/O) and 
b. data storage. Classes I use for the overall logic of the 
program (with the usual suspects like controller, model, proxies 
...) But I'm always willing to learn new tricks (this dog is not 
too old yet!).
Sep 30 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-09-30 16:08, Dicebot wrote:


 I don't think there is a common consensus here. Don't even think it is
 possible. Doom of proper multi-paradigm language :)


At least a few point can be made, like:

* For reference types use classes, or use structs with reference counting
* For sub typing use classes, or use structs with alias this
* For polymorphic types use classes
* For value types use structs
* For deterministic destruction use structs

-- 
/Jacob Carlborg
Sep 30 2013
parent "Chris" <wendlec tcd.ie> writes:
On Monday, 30 September 2013 at 17:19:44 UTC, Jacob Carlborg 
wrote:

 On 2013-09-30 16:08, Dicebot wrote:


 I don't think there is a common consensus here. Don't even 
 think it is
 possible. Doom of proper multi-paradigm language :)


 At least a few point can be made, like:

 * For reference types use classes, or use structs with 
 reference counting
 * For sub typing use classes, or use structs with alias this
 * For polymorphic types use classes
 * For value types use structs
 * For deterministic destruction use structs


That's a good point to start. The points above could be fleshed 
out in a tutorial or article. Another good section would be the 
pitfalls involved in using structs. I think it was H. S. Teoh who 
mentioned a subtle bug due to a missing ref or something like 
that.
Sep 30 2013
prev sibling next sibling parent reply "Brad Anderson" <eco gnuk.net> writes:
On Saturday, 28 September 2013 at 16:39:27 UTC, simendsjo wrote:

 I've been working on a more or less complete rewrite of the 
 mysql-native module.


I don't have any specific comments.  Just want to say that Steve 
Teale originally wrote the module as part of his std.database 
efforts.  I think it would be nice if any redesign done would aim 
for eventual phobos inclusion.
Sep 30 2013
parent reply "simendsjo" <simendsjo gmail.com> writes:
On Monday, 30 September 2013 at 16:27:16 UTC, Brad Anderson wrote:

 On Saturday, 28 September 2013 at 16:39:27 UTC, simendsjo wrote:

 I've been working on a more or less complete rewrite of the 
 mysql-native module.


 I don't have any specific comments.  Just want to say that 
 Steve Teale originally wrote the module as part of his 
 std.database efforts.  I think it would be nice if any redesign 
 done would aim for eventual phobos inclusion.


I don't agree that this library should aim for any inclusion in 
phobos. I plan on exposing the MySQL API as closely as possible 
and a couple of higher level convenience functions. The 
std.database would have to be database agnostic, and should be a 
library interfacing to mysql-native, postgres etc.

We should contact the author of ddbc for input: 
https://github.com/buggins/ddbc
Sep 30 2013
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-09-30 20:14, simendsjo wrote:


 I don't agree that this library should aim for any inclusion in phobos.
 I plan on exposing the MySQL API as closely as possible and a couple of
 higher level convenience functions. The std.database would have to be
 database agnostic, and should be a library interfacing to mysql-native,
 postgres etc.


Yes, std.database should be database agnostic. But that doesn't mean we 
can't include one or a couple of database adapters. For example, Ruby on 
Rails contains a couple of database adapters, but not for all databases. 
It has no adapter for SQL Server, for that you need a separate gem/plugin.

An adapter for MySQL would probably be good to include. Even better, an 
adapter for SQLite. Perhaps we can include the whole SQLite source code, 
since it's public domain.

-- 
/Jacob Carlborg
Oct 01 2013
prev sibling next sibling parent reply "ilya-stromberg" <ilya-stromberg-2009 yandex.ru> writes:
On Monday, 30 September 2013 at 18:14:13 UTC, simendsjo wrote:

 On Monday, 30 September 2013 at 16:27:16 UTC, Brad Anderson 
 wrote:

 On Saturday, 28 September 2013 at 16:39:27 UTC, simendsjo 
 wrote:

 I've been working on a more or less complete rewrite of the 
 mysql-native module.


 I don't have any specific comments.  Just want to say that 
 Steve Teale originally wrote the module as part of his 
 std.database efforts.  I think it would be nice if any 
 redesign done would aim for eventual phobos inclusion.


 I don't agree that this library should aim for any inclusion in 
 phobos. I plan on exposing the MySQL API as closely as possible 
 and a couple of higher level convenience functions. The 
 std.database would have to be database agnostic, and should be 
 a library interfacing to mysql-native, postgres etc.

 We should contact the author of ddbc for input: 
 https://github.com/buggins/ddbc


No, you are wrong. We need both low-level and high-level API,
Low-level API like mysql-native driver should present a specific 
database as much as possible and provide access to all database 
features.
High-level API like ORM HibernateD must provide common API for 
all databases, but it can't present all database specific 
features due database differences.
So, I can use low-level API if I need access to all database 
specific features. I can use high-level API if I need common API 
for all databases.
Oct 01 2013
parent "simendsjo" <simendsjo gmail.com> writes:
On Tuesday, 1 October 2013 at 08:05:29 UTC, ilya-stromberg wrote:

 On Monday, 30 September 2013 at 18:14:13 UTC, simendsjo wrote:

 On Monday, 30 September 2013 at 16:27:16 UTC, Brad Anderson 
 wrote:

 On Saturday, 28 September 2013 at 16:39:27 UTC, simendsjo 
 wrote:

 I've been working on a more or less complete rewrite of the 
 mysql-native module.


 I don't have any specific comments.  Just want to say that 
 Steve Teale originally wrote the module as part of his 
 std.database efforts.  I think it would be nice if any 
 redesign done would aim for eventual phobos inclusion.


 I don't agree that this library should aim for any inclusion 
 in phobos. I plan on exposing the MySQL API as closely as 
 possible and a couple of higher level convenience functions. 
 The std.database would have to be database agnostic, and 
 should be a library interfacing to mysql-native, postgres etc.

 We should contact the author of ddbc for input: 
 https://github.com/buggins/ddbc


 No, you are wrong. We need both low-level and high-level API,
 Low-level API like mysql-native driver should present a 
 specific database as much as possible and provide access to all 
 database features.
 High-level API like ORM HibernateD must provide common API for 
 all databases, but it can't present all database specific 
 features due database differences.
 So, I can use low-level API if I need access to all database 
 specific features. I can use high-level API if I need common 
 API for all databases.


That was my point.
Oct 01 2013
prev sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Mon, 30 Sep 2013 20:14:12 +0200
"simendsjo" <simendsjo gmail.com> wrote:


 On Monday, 30 September 2013 at 16:27:16 UTC, Brad Anderson wrote:

 On Saturday, 28 September 2013 at 16:39:27 UTC, simendsjo wrote:

 I've been working on a more or less complete rewrite of the 
 mysql-native module.


 I don't have any specific comments.  Just want to say that 
 Steve Teale originally wrote the module as part of his 
 std.database efforts.  I think it would be nice if any redesign 
 done would aim for eventual phobos inclusion.


 
 I don't agree that this library should aim for any inclusion in 
 phobos.


While certainly I have no objection to database stuff in Phobos, I
don't think it's particularly important now that we have dub.
Steve Teale's intension of getting it into Phobos predated anything
like that, so Phobos needed to be "kitchen sick". Not sure that's quite
as applicable anymore.
Oct 01 2013
prev sibling parent reply ollie <ollie home.net> writes:
On Sat, 28 Sep 2013 18:38:47 +0200, simendsjo wrote:


 I've been working on a more or less complete rewrite of the 
 mysql-native module.
 


I think this is a great first step. Code is more readable, easier
to follow (compared to the pointer stuff in the original).

Code has some distinctly 64bit parts because Length Encoded Integer
can be as large as 48bits and consume functions use a mix of size_t
and (u)long. Code doesn't compile as is on 32bit machine (windows).
I have changed some (u)longs to size_t and put in some cast(size_t) on
the assumption that on a 32bit machine those values won't overflow a
uint and on 64bit machine the will be (u)long as written. It compiles
and works for some basic queries that I use. This is a hack, not a
solution to the problem.

Also, the socket reads are asserting because windows reads partial
packets. I change it to read until a full packet is read or an error
or disconnect is returned.

simendsjo, you have done a great job pushing this forward. Would you
consider putting it on your github account to receive some patches?

ollie
Oct 08 2013
parent "simendsjo" <simendsjo gmail.com> writes:
On Tuesday, 8 October 2013 at 17:47:02 UTC, ollie wrote:

 On Sat, 28 Sep 2013 18:38:47 +0200, simendsjo wrote:


 I've been working on a more or less complete rewrite of the 
 mysql-native module.
 


 I think this is a great first step. Code is more readable, 
 easier
 to follow (compared to the pointer stuff in the original).



I removed most of the pointer stuff in my initial cleanup, but 
there has still
been plenty of room for improvements.



 Code has some distinctly 64bit parts because Length Encoded 
 Integer
 can be as large as 48bits and consume functions use a mix of 
 size_t
 and (u)long. Code doesn't compile as is on 32bit machine 
 (windows).
 I have changed some (u)longs to size_t and put in some 
 cast(size_t) on
 the assumption that on a 32bit machine those values won't 
 overflow a
 uint and on 64bit machine the will be (u)long as written. It 
 compiles
 and works for some basic queries that I use. This is a hack, 
 not a
 solution to the problem.



I might have mixed up some of these I guess.. Using size_t in the 
protocol
specific code is probably wrong. I'll have to go through this in 
detail.



 Also, the socket reads are asserting because windows reads 
 partial
 packets. I change it to read until a full packet is read or an 
 error
 or disconnect is returned.



Auch.. Hope this fix solves it once and for all - otherwise there 
will probably
be a lot of subtle breakage in the code.


 simendsjo, you have done a great job pushing this forward. 
 Would you
 consider putting it on your github account to receive some 
 patches?

 ollie



https://github.com/simendsjo/mysqln

I think the only thing I changed from the upload is bugfixes in 
parameter
binding and added a couple of integration tests. I would have 
hoped to solve
a lot of these things before uploading the code, but if you wish 
to contribute,
I'm not going to stop you :)

There is still plenty to do though (some has been raised in this 
thread):
o 32-bit support
o Windows support
   (and OSX and FreeBSD)
o Documentation
o Examples
o Unittests
o More integration tests
o Implement stored procedures
   (and replication..?)
o SSL
o Remove type special handling (TINYINT(1) -> bool and perhaps 
others)
o Range interface for COM_QUERY and COM_STMT_EXECUTE that returns 
results.
o .. Which means ResultSet also needs a range interface and some 
way to
   construct it by taking all raws eagerly. I'm thinking 
"queryLazy" vs "query".
o COM_QUERY and COM_STMT_EXECUTE should be classes with close() 
methods to stop
   the execution. This will probably mean we need some data 
purging or
   something, and it has to be an error to start a new command 
before a previous
   has finished.
o Rewrite parameter binding code - it is very ugly and hard to 
understand.
   The old way was to store Variants and use typeid.toString to 
find out how to
   bind the parameter. I've rewritten this to store the parameter 
in a packet
   right away so it works with literals, const etc. The code is 
very ugly
   though. See packets.ComStmtPrepare
o There are stuff I haven't figured out in the prepared 
statements yet.
   Cursors and new_params_bound comes to mind.
o There might be unused functions in packet_helpers, but I'd like 
full
   integration-tests before removing anything.
o .. And probably many other stuff
Oct 09 2013