www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - [RFC] - mysql-native rewrite

reply "simendsjo" <simendsjo gmail.com> writes:
I've been working on a more or less complete rewrite of the 
mysql-native module.

The main focus has been on making the source better decoupled and 
more maintainable.
In the process, I've changed the API too...

The existing production code can be found here: 
https://github.com/rejectedsoftware/mysql-native

New code: 
http://simendsjo.me/files/simendsjo/mysqln-rewrite/mysqln-rewrite.7z
Documentation: 
http://simendsjo.me/files/simendsjo/mysqln-rewrite/docs/

The files:
* adhoc.d - Execute simple queries. This is meant to be the new 
high-level public API
* commands.d - Lower level public API
* result.d - Common interface for both Text and Binary MySQL 
results
* connection.d - Phobos/Vibe.d connection
* db.d - Vibe.d connection pool

The rest are just internal files.

Todo:
* Implement all MySQL types
* Prepared statement release/purge
* Lazily fetch data
* Fix bugs in prepared statement binding
* Implement SEND_LONG_DATA
* More integrationtests and unittests

Most of the above would be pretty simple to implement.

I'm very uncertain about several aspects of my design:
* Class vs. struct
* Returned values from MySQL - e.g. should SELECT TRUE return 
long as it does in MySQL, or should we interpret it as bool
* Probably a lot I don't remember right now :)

Any comments would be very helpful.
..Or in D terms: DESTROY!
Sep 28 2013
next sibling parent "Gary Willoughby" <dev nomad.so> writes:
On Saturday, 28 September 2013 at 16:39:27 UTC, simendsjo wrote:
 I've been working on a more or less complete rewrite of the 
 mysql-native module.

 The main focus has been on making the source better decoupled 
 and more maintainable.
 In the process, I've changed the API too...

 The existing production code can be found here: 
 https://github.com/rejectedsoftware/mysql-native

 New code: 
 http://simendsjo.me/files/simendsjo/mysqln-rewrite/mysqln-rewrite.7z
 Documentation: 
 http://simendsjo.me/files/simendsjo/mysqln-rewrite/docs/

 The files:
 * adhoc.d - Execute simple queries. This is meant to be the new 
 high-level public API
 * commands.d - Lower level public API
 * result.d - Common interface for both Text and Binary MySQL 
 results
 * connection.d - Phobos/Vibe.d connection
 * db.d - Vibe.d connection pool

 The rest are just internal files.

 Todo:
 * Implement all MySQL types
 * Prepared statement release/purge
 * Lazily fetch data
 * Fix bugs in prepared statement binding
 * Implement SEND_LONG_DATA
 * More integrationtests and unittests

 Most of the above would be pretty simple to implement.

 I'm very uncertain about several aspects of my design:
 * Class vs. struct
 * Returned values from MySQL - e.g. should SELECT TRUE return 
 long as it does in MySQL, or should we interpret it as bool
 * Probably a lot I don't remember right now :)

 Any comments would be very helpful.
 ..Or in D terms: DESTROY!

This sounds great! I use this library a lot so i can probably give you some feedback when i have more time. I'm about to go out now but i'll try to take a look tomorrow.
Sep 28 2013
prev sibling next sibling parent reply "Gary Willoughby" <dev nomad.so> writes:
On Saturday, 28 September 2013 at 16:39:27 UTC, simendsjo wrote:
 I'm very uncertain about several aspects of my design:
 * Class vs. struct
 * Returned values from MySQL - e.g. should SELECT TRUE return 
 long as it does in MySQL, or should we interpret it as bool
 * Probably a lot I don't remember right now :)

 Any comments would be very helpful.
 ..Or in D terms: DESTROY!

Ok, i've taken a quick look and like where this is going. I'll try and give you a little guide to what i would do regarding the above points.
 * Class vs. struct

I tend to only use structs where the type is purely a data type. Something like a record or a type that can be manifested in different ways (such as an IP address). If i need to model a real world object like an engine or book, etc., i use a class.
 * Returned values from MySQL - e.g. should SELECT TRUE return 
 long as it does in MySQL, or should we interpret it as bool

I would return what mysql yields. This would make sure this framework is not doing to much. Converting ints to bools would be the next layer's job (DBAL, ORM, etc.). I have two more suggestions that i think is critical to a project such as this and that's documentation and unit tests. Please from the start thoroughly annotate everything with ddoc to generate nice html docs later. I know this can be a pain when designing and developing an API but is absolutely necessary. If you leave documentation until the end it never gets done! Write unit tests for everything. I've found that if you find it hard to write a unit test for any 'unit', etc then it's too tightly coupled and probably doing too much. Practise good SOLID design principles and unit tests should be easy to write. Not only that but unit tests provide developers with a good understanding of the public interface. http://en.wikipedia.org/wiki/SOLID_(object-oriented_design)
Sep 29 2013
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-09-30 13:20, simendsjo wrote:

 Yeah. We need to choose:
 1) Starting a new command while another is in flight is an error
     You need to close explicitly if the command isn't finished
 2) If another command has been started, it's er error to continue
 iteration of a previous command.

 I'm in favor of 1).

I would go with one as well.
 Yeah. BOOL is an alias for TINYINT(1). I think it's fair to always
 assume TINYINT(1) is bool. For other types I'm not quite sure though..
 "SELECT 1" is a LONGLONG, but in D a literal is int unless specified
 otherwise. So there is a mismatch here.

Use the type in D which has the closets match and have the same length. Type conversions can be made later in a higher level API. -- /Jacob Carlborg
Sep 30 2013
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-09-30 15:07, Gary Willoughby wrote:

 Please don't. The db library should only return what mysql returns. Let
 the next layer handle casting types.

Agree. -- /Jacob Carlborg
Sep 30 2013
prev sibling next sibling parent "simendsjo" <simendsjo gmail.com> writes:
On Sunday, 29 September 2013 at 15:26:19 UTC, Gary Willoughby 
wrote:
 On Saturday, 28 September 2013 at 16:39:27 UTC, simendsjo wrote:
 I'm very uncertain about several aspects of my design:
 * Class vs. struct
 * Returned values from MySQL - e.g. should SELECT TRUE return 
 long as it does in MySQL, or should we interpret it as bool
 * Probably a lot I don't remember right now :)

 Any comments would be very helpful.
 ..Or in D terms: DESTROY!

Ok, i've taken a quick look and like where this is going. I'll try and give you a little guide to what i would do regarding the above points.
 * Class vs. struct

I tend to only use structs where the type is purely a data type. Something like a record or a type that can be manifested in different ways (such as an IP address). If i need to model a real world object like an engine or book, etc., i use a class.

I don't think it's that simple in this case. When I implement lazy fetching, both methods have it's advantages and disadvantages. MySQL doesn't allow multiplexing on a connection. This means a command must complete before issuing another. If reading rows lazily and then issuing a new command, the choice is to either disallow the new command, or invalidate the previous. The simple way would be to just start a new command and invalidate the previous, but is this the best way? If we choose to disallow new commands, that means the user have to explicitly close a lazy command. If using classes, we can safely have several instances for a command (is this neccessary?), but then the destructor wont't be a safe bet, and the user have to call close. If implemented as a struct, we have to disallow copying. So... I really don't know what the best design would be.
 * Returned values from MySQL - e.g. should SELECT TRUE return 
 long as it does in MySQL, or should we interpret it as bool

I would return what mysql yields. This would make sure this framework is not doing to much. Converting ints to bools would be the next layer's job (DBAL, ORM, etc.).

The problem is that MySQL is inconsistent here, and it depends on if it's a field or a constant. SELECT TRUE is not the same as SELECT boolean_field. Also, it seems every constant integer is returned as LONGLONG. But.. The fields include a length property..
 I have two more suggestions that i think is critical to a 
 project such as this and that's documentation and unit tests.

 Please from the start thoroughly annotate everything with ddoc 
 to generate nice html docs later. I know this can be a pain 
 when designing and developing an API but is absolutely 
 necessary. If you leave documentation until the end it never 
 gets done!

Yeah, the documentation isn't very good :/ Thought I'd get feedback on the API before spending a lot of time documenting it.
 Write unit tests for everything. I've found that if you find it 
 hard to write a unit test for any 'unit', etc then it's too 
 tightly coupled and probably doing too much. Practise good 
 SOLID design principles and unit tests should be easy to write. 
 Not only that but unit tests provide developers with a good 
 understanding of the public interface.

 http://en.wikipedia.org/wiki/SOLID_(object-oriented_design)

Most of the code is easy to unittest, but for now I've just relied on integration tests against a database. I'll improve on the situation.
Sep 29 2013
prev sibling next sibling parent "Gary Willoughby" <dev nomad.so> writes:
On Sunday, 29 September 2013 at 15:42:10 UTC, simendsjo wrote:
 I don't think it's that simple in this case. When I implement 
 lazy fetching, both methods have it's advantages and 
 disadvantages.
 MySQL doesn't allow multiplexing on a connection. This means a 
 command must complete before issuing another. If reading rows 
 lazily and then issuing a new command, the choice is to either 
 disallow the new command, or invalidate the previous. The 
 simple way would be to just start a new command and invalidate 
 the previous, but is this the best way? If we choose to 
 disallow new commands, that means the user have to explicitly 
 close a lazy command. If using classes, we can safely have 
 several instances for a command (is this neccessary?), but then 
 the destructor wont't be a safe bet, and the user have to call 
 close.
 If implemented as a struct, we have to disallow copying.

 So... I really don't know what the best design would be.

I reckon go with Command being a class and invalidate previous lazy loading on new commands being issued, you are after all issuing a *new* command. If multiplexing is needed a new connection can be made.
Sep 29 2013
prev sibling next sibling parent "Kagamin" <spam here.lot> writes:
On Sunday, 29 September 2013 at 15:42:10 UTC, simendsjo wrote:
 the previous, but is this the best way? If we choose to 
 disallow new commands, that means the user have to explicitly 
 close a lazy command.

If a command isn't closed as soon as possible, it sounds like a resource leak. If you silently cancel previous query, it can hide logical bug.
 If using classes, we can safely have several instances for a 
 command (is this neccessary?)

If you execute the same query repeatedly, there's no need to build the same command every time.
 The problem is that MySQL is inconsistent here, and it depends 
 on if it's a field or a constant. SELECT TRUE is not the same 
 as SELECT boolean_field.
 Also, it seems every constant integer is returned as LONGLONG. 
 But.. The fields include a length property..

Is there some hint it's true and not something else?
Sep 30 2013
prev sibling next sibling parent "simendsjo" <simendsjo gmail.com> writes:
On Monday, 30 September 2013 at 10:34:25 UTC, Kagamin wrote:
 On Sunday, 29 September 2013 at 15:42:10 UTC, simendsjo wrote:
 the previous, but is this the best way? If we choose to 
 disallow new commands, that means the user have to explicitly 
 close a lazy command.

If a command isn't closed as soon as possible, it sounds like a resource leak. If you silently cancel previous query, it can hide logical bug.

Yeah. We need to choose: 1) Starting a new command while another is in flight is an error You need to close explicitly if the command isn't finished 2) If another command has been started, it's er error to continue iteration of a previous command. I'm in favor of 1).
 If using classes, we can safely have several instances for a 
 command (is this neccessary?)

If you execute the same query repeatedly, there's no need to build the same command every time.

That's what prepared statements are for. If you don't even have parameters, it's just 9 bytes (4 of these bytes is an undocumented field called iteration_count..).
 The problem is that MySQL is inconsistent here, and it depends 
 on if it's a field or a constant. SELECT TRUE is not the same 
 as SELECT boolean_field.
 Also, it seems every constant integer is returned as LONGLONG. 
 But.. The fields include a length property..

Is there some hint it's true and not something else?

Yeah. BOOL is an alias for TINYINT(1). I think it's fair to always assume TINYINT(1) is bool. For other types I'm not quite sure though.. "SELECT 1" is a LONGLONG, but in D a literal is int unless specified otherwise. So there is a mismatch here.
Sep 30 2013
prev sibling next sibling parent "Kagamin" <spam here.lot> writes:
On Monday, 30 September 2013 at 11:20:23 UTC, simendsjo wrote:
 Yeah. We need to choose:
 1) Starting a new command while another is in flight is an error
    You need to close explicitly if the command isn't finished
 2) If another command has been started, it's er error to 
 continue iteration of a previous command.

 I'm in favor of 1).

That's what .net does. Most of the time it indicates a resource leak.
 Yeah. BOOL is an alias for TINYINT(1). I think it's fair to 
 always assume TINYINT(1) is bool. For other types I'm not quite 
 sure though.. "SELECT 1" is a LONGLONG, but in D a literal is 
 int unless specified otherwise. So there is a mismatch here.

int64 is the right type for a database integer, int32 is too small. After all, it's a database.
Sep 30 2013
prev sibling next sibling parent "simendsjo" <simendsjo gmail.com> writes:
On Monday, 30 September 2013 at 11:58:48 UTC, Kagamin wrote:
 On Monday, 30 September 2013 at 11:20:23 UTC, simendsjo wrote:
 Yeah. We need to choose:
 1) Starting a new command while another is in flight is an 
 error
   You need to close explicitly if the command isn't finished
 2) If another command has been started, it's er error to 
 continue iteration of a previous command.

 I'm in favor of 1).

That's what .net does. Most of the time it indicates a resource leak.

Ok. I'll add the same semantics.
 Yeah. BOOL is an alias for TINYINT(1). I think it's fair to 
 always assume TINYINT(1) is bool. For other types I'm not 
 quite sure though.. "SELECT 1" is a LONGLONG, but in D a 
 literal is int unless specified otherwise. So there is a 
 mismatch here.

int64 is the right type for a database integer, int32 is too small. After all, it's a database.

I'll just add a hack for TINYINT(1) -> bool and keep everything else MySQL specific. Have to admit that int32 has been plenty for me in the past. Never worked on databases with > 200M rows.
Sep 30 2013
prev sibling next sibling parent "Kagamin" <spam here.lot> writes:
It's not only size. A popular use case is a log: you write lots 
of entries and delete old ones, total size doesn't grow, but the 
identifiers...
Sep 30 2013
prev sibling next sibling parent "Gary Willoughby" <dev nomad.so> writes:
On Monday, 30 September 2013 at 12:05:49 UTC, simendsjo wrote:
 I'll just add a hack for TINYINT(1) -> bool and keep everything 
 else MySQL specific.

Please don't. The db library should only return what mysql returns. Let the next layer handle casting types.
 Have to admit that int32 has been plenty for me in the past. 
 Never worked on databases with > 200M rows.

I routinely use db's with enormous datasets 500M+ and use BIGINTS a lot. If i select a BIGINT column i'd expect a long back.
Sep 30 2013
prev sibling next sibling parent "Gary Willoughby" <dev nomad.so> writes:
On Monday, 30 September 2013 at 11:58:48 UTC, Kagamin wrote:
 int64 is the right type for a database integer, int32 is too 
 small. After all, it's a database.

All mysql types should be handled by using the correct type in D. e.g. if i select an INT i want a D int. If i select a BINGINT i want a D long, etc.
Sep 30 2013
prev sibling next sibling parent reply "Chris" <wendlec tcd.ie> writes:
On Saturday, 28 September 2013 at 16:39:27 UTC, simendsjo wrote:
 I've been working on a more or less complete rewrite of the 
 mysql-native module.

 The main focus has been on making the source better decoupled 
 and more maintainable.
 In the process, I've changed the API too...

 The existing production code can be found here: 
 https://github.com/rejectedsoftware/mysql-native

 New code: 
 http://simendsjo.me/files/simendsjo/mysqln-rewrite/mysqln-rewrite.7z
 Documentation: 
 http://simendsjo.me/files/simendsjo/mysqln-rewrite/docs/

 The files:
 * adhoc.d - Execute simple queries. This is meant to be the new 
 high-level public API
 * commands.d - Lower level public API
 * result.d - Common interface for both Text and Binary MySQL 
 results
 * connection.d - Phobos/Vibe.d connection
 * db.d - Vibe.d connection pool

 The rest are just internal files.

 Todo:
 * Implement all MySQL types
 * Prepared statement release/purge
 * Lazily fetch data
 * Fix bugs in prepared statement binding
 * Implement SEND_LONG_DATA
 * More integrationtests and unittests

 Most of the above would be pretty simple to implement.

 I'm very uncertain about several aspects of my design:
 * Class vs. struct
 * Returned values from MySQL - e.g. should SELECT TRUE return 
 long as it does in MySQL, or should we interpret it as bool
 * Probably a lot I don't remember right now :)

 Any comments would be very helpful.
 ..Or in D terms: DESTROY!

It would be nice, if someone with experience wrote a short article / tutorial about when to use structs and when to use classes, where the pitfalls are etc. It is not always easy to decide and a rough guide would be great. As of now I'm using structs and ranges only for reformatting input. I'm not sure, however, if it is a good idea to go beyond this (cf. different reference semantics, polymorphism etc.)
Sep 30 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-09-30 16:08, Dicebot wrote:

 I don't think there is a common consensus here. Don't even think it is
 possible. Doom of proper multi-paradigm language :)

At least a few point can be made, like: * For reference types use classes, or use structs with reference counting * For sub typing use classes, or use structs with alias this * For polymorphic types use classes * For value types use structs * For deterministic destruction use structs -- /Jacob Carlborg
Sep 30 2013
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Monday, 30 September 2013 at 14:00:10 UTC, Chris wrote:
 It would be nice, if someone with experience wrote a short 
 article / tutorial about when to use structs and when to use 
 classes, where the pitfalls are etc.

I don't think there is a common consensus here. Don't even think it is possible. Doom of proper multi-paradigm language :)
Sep 30 2013
prev sibling next sibling parent "Chris" <wendlec tcd.ie> writes:
On Monday, 30 September 2013 at 14:08:12 UTC, Dicebot wrote:
 On Monday, 30 September 2013 at 14:00:10 UTC, Chris wrote:
 It would be nice, if someone with experience wrote a short 
 article / tutorial about when to use structs and when to use 
 classes, where the pitfalls are etc.

I don't think there is a common consensus here. Don't even think it is possible. Doom of proper multi-paradigm language :)

Well, if people shared their experience with ranges / structs vs. classes, it would be a good reference point for other programmers. There might not be any real rules, but rough guidelines would help, e.g. "If you want to achieve X, structs are better, because blah blah. But if you want to do Y, there might be a problem with structs. Better use classes." Maybe structs are better suited for parallel processing and multi threading (value vs reference), or maybe not? Just a collection of little hints would make life simpler. I'm sure someone has shot him/herself seriously in the foot using either structs or classes. I use structs mainly for a. formatting and reformatting (I/O) and b. data storage. Classes I use for the overall logic of the program (with the usual suspects like controller, model, proxies ...) But I'm always willing to learn new tricks (this dog is not too old yet!).
Sep 30 2013
prev sibling next sibling parent reply "Brad Anderson" <eco gnuk.net> writes:
On Saturday, 28 September 2013 at 16:39:27 UTC, simendsjo wrote:
 I've been working on a more or less complete rewrite of the 
 mysql-native module.

I don't have any specific comments. Just want to say that Steve Teale originally wrote the module as part of his std.database efforts. I think it would be nice if any redesign done would aim for eventual phobos inclusion.
Sep 30 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-09-30 20:14, simendsjo wrote:

 I don't agree that this library should aim for any inclusion in phobos.
 I plan on exposing the MySQL API as closely as possible and a couple of
 higher level convenience functions. The std.database would have to be
 database agnostic, and should be a library interfacing to mysql-native,
 postgres etc.

Yes, std.database should be database agnostic. But that doesn't mean we can't include one or a couple of database adapters. For example, Ruby on Rails contains a couple of database adapters, but not for all databases. It has no adapter for SQL Server, for that you need a separate gem/plugin. An adapter for MySQL would probably be good to include. Even better, an adapter for SQLite. Perhaps we can include the whole SQLite source code, since it's public domain. -- /Jacob Carlborg
Oct 01 2013
prev sibling next sibling parent "simendsjo" <simendsjo gmail.com> writes:
On Monday, 30 September 2013 at 13:33:09 UTC, Jacob Carlborg 
wrote:
 On 2013-09-30 15:07, Gary Willoughby wrote:

 Please don't. The db library should only return what mysql 
 returns. Let
 the next layer handle casting types.

Agree.

Seems everyone agrees on this. In that case, I won't do any kind of processing of the types.
Sep 30 2013
prev sibling next sibling parent "simendsjo" <simendsjo gmail.com> writes:
On Monday, 30 September 2013 at 16:27:16 UTC, Brad Anderson wrote:
 On Saturday, 28 September 2013 at 16:39:27 UTC, simendsjo wrote:
 I've been working on a more or less complete rewrite of the 
 mysql-native module.

I don't have any specific comments. Just want to say that Steve Teale originally wrote the module as part of his std.database efforts. I think it would be nice if any redesign done would aim for eventual phobos inclusion.

I don't agree that this library should aim for any inclusion in phobos. I plan on exposing the MySQL API as closely as possible and a couple of higher level convenience functions. The std.database would have to be database agnostic, and should be a library interfacing to mysql-native, postgres etc. We should contact the author of ddbc for input: https://github.com/buggins/ddbc
Sep 30 2013
prev sibling next sibling parent "Chris" <wendlec tcd.ie> writes:
On Monday, 30 September 2013 at 17:19:44 UTC, Jacob Carlborg 
wrote:
 On 2013-09-30 16:08, Dicebot wrote:

 I don't think there is a common consensus here. Don't even 
 think it is
 possible. Doom of proper multi-paradigm language :)

At least a few point can be made, like: * For reference types use classes, or use structs with reference counting * For sub typing use classes, or use structs with alias this * For polymorphic types use classes * For value types use structs * For deterministic destruction use structs

That's a good point to start. The points above could be fleshed out in a tutorial or article. Another good section would be the pitfalls involved in using structs. I think it was H. S. Teoh who mentioned a subtle bug due to a missing ref or something like that.
Sep 30 2013
prev sibling next sibling parent "ilya-stromberg" <ilya-stromberg-2009 yandex.ru> writes:
On Monday, 30 September 2013 at 18:14:13 UTC, simendsjo wrote:
 On Monday, 30 September 2013 at 16:27:16 UTC, Brad Anderson 
 wrote:
 On Saturday, 28 September 2013 at 16:39:27 UTC, simendsjo 
 wrote:
 I've been working on a more or less complete rewrite of the 
 mysql-native module.

I don't have any specific comments. Just want to say that Steve Teale originally wrote the module as part of his std.database efforts. I think it would be nice if any redesign done would aim for eventual phobos inclusion.

I don't agree that this library should aim for any inclusion in phobos. I plan on exposing the MySQL API as closely as possible and a couple of higher level convenience functions. The std.database would have to be database agnostic, and should be a library interfacing to mysql-native, postgres etc. We should contact the author of ddbc for input: https://github.com/buggins/ddbc

No, you are wrong. We need both low-level and high-level API, Low-level API like mysql-native driver should present a specific database as much as possible and provide access to all database features. High-level API like ORM HibernateD must provide common API for all databases, but it can't present all database specific features due database differences. So, I can use low-level API if I need access to all database specific features. I can use high-level API if I need common API for all databases.
Oct 01 2013
prev sibling next sibling parent "simendsjo" <simendsjo gmail.com> writes:
On Tuesday, 1 October 2013 at 08:05:29 UTC, ilya-stromberg wrote:
 On Monday, 30 September 2013 at 18:14:13 UTC, simendsjo wrote:
 On Monday, 30 September 2013 at 16:27:16 UTC, Brad Anderson 
 wrote:
 On Saturday, 28 September 2013 at 16:39:27 UTC, simendsjo 
 wrote:
 I've been working on a more or less complete rewrite of the 
 mysql-native module.

I don't have any specific comments. Just want to say that Steve Teale originally wrote the module as part of his std.database efforts. I think it would be nice if any redesign done would aim for eventual phobos inclusion.

I don't agree that this library should aim for any inclusion in phobos. I plan on exposing the MySQL API as closely as possible and a couple of higher level convenience functions. The std.database would have to be database agnostic, and should be a library interfacing to mysql-native, postgres etc. We should contact the author of ddbc for input: https://github.com/buggins/ddbc

No, you are wrong. We need both low-level and high-level API, Low-level API like mysql-native driver should present a specific database as much as possible and provide access to all database features. High-level API like ORM HibernateD must provide common API for all databases, but it can't present all database specific features due database differences. So, I can use low-level API if I need access to all database specific features. I can use high-level API if I need common API for all databases.

That was my point.
Oct 01 2013
prev sibling next sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Mon, 30 Sep 2013 20:14:12 +0200
"simendsjo" <simendsjo gmail.com> wrote:

 On Monday, 30 September 2013 at 16:27:16 UTC, Brad Anderson wrote:
 On Saturday, 28 September 2013 at 16:39:27 UTC, simendsjo wrote:
 I've been working on a more or less complete rewrite of the 
 mysql-native module.

I don't have any specific comments. Just want to say that Steve Teale originally wrote the module as part of his std.database efforts. I think it would be nice if any redesign done would aim for eventual phobos inclusion.

I don't agree that this library should aim for any inclusion in phobos.

While certainly I have no objection to database stuff in Phobos, I don't think it's particularly important now that we have dub. Steve Teale's intension of getting it into Phobos predated anything like that, so Phobos needed to be "kitchen sick". Not sure that's quite as applicable anymore.
Oct 01 2013
prev sibling next sibling parent ollie <ollie home.net> writes:
On Sat, 28 Sep 2013 18:38:47 +0200, simendsjo wrote:

 I've been working on a more or less complete rewrite of the 
 mysql-native module.
 

I think this is a great first step. Code is more readable, easier to follow (compared to the pointer stuff in the original). Code has some distinctly 64bit parts because Length Encoded Integer can be as large as 48bits and consume functions use a mix of size_t and (u)long. Code doesn't compile as is on 32bit machine (windows). I have changed some (u)longs to size_t and put in some cast(size_t) on the assumption that on a 32bit machine those values won't overflow a uint and on 64bit machine the will be (u)long as written. It compiles and works for some basic queries that I use. This is a hack, not a solution to the problem. Also, the socket reads are asserting because windows reads partial packets. I change it to read until a full packet is read or an error or disconnect is returned. simendsjo, you have done a great job pushing this forward. Would you consider putting it on your github account to receive some patches? ollie
Oct 08 2013
prev sibling parent "simendsjo" <simendsjo gmail.com> writes:
On Tuesday, 8 October 2013 at 17:47:02 UTC, ollie wrote:
 On Sat, 28 Sep 2013 18:38:47 +0200, simendsjo wrote:

 I've been working on a more or less complete rewrite of the 
 mysql-native module.
 

I think this is a great first step. Code is more readable, easier to follow (compared to the pointer stuff in the original).

I removed most of the pointer stuff in my initial cleanup, but there has still been plenty of room for improvements.
 Code has some distinctly 64bit parts because Length Encoded 
 Integer
 can be as large as 48bits and consume functions use a mix of 
 size_t
 and (u)long. Code doesn't compile as is on 32bit machine 
 (windows).
 I have changed some (u)longs to size_t and put in some 
 cast(size_t) on
 the assumption that on a 32bit machine those values won't 
 overflow a
 uint and on 64bit machine the will be (u)long as written. It 
 compiles
 and works for some basic queries that I use. This is a hack, 
 not a
 solution to the problem.

I might have mixed up some of these I guess.. Using size_t in the protocol specific code is probably wrong. I'll have to go through this in detail.
 Also, the socket reads are asserting because windows reads 
 partial
 packets. I change it to read until a full packet is read or an 
 error
 or disconnect is returned.

Auch.. Hope this fix solves it once and for all - otherwise there will probably be a lot of subtle breakage in the code.
 simendsjo, you have done a great job pushing this forward. 
 Would you
 consider putting it on your github account to receive some 
 patches?

 ollie

https://github.com/simendsjo/mysqln I think the only thing I changed from the upload is bugfixes in parameter binding and added a couple of integration tests. I would have hoped to solve a lot of these things before uploading the code, but if you wish to contribute, I'm not going to stop you :) There is still plenty to do though (some has been raised in this thread): o 32-bit support o Windows support (and OSX and FreeBSD) o Documentation o Examples o Unittests o More integration tests o Implement stored procedures (and replication..?) o SSL o Remove type special handling (TINYINT(1) -> bool and perhaps others) o Range interface for COM_QUERY and COM_STMT_EXECUTE that returns results. o .. Which means ResultSet also needs a range interface and some way to construct it by taking all raws eagerly. I'm thinking "queryLazy" vs "query". o COM_QUERY and COM_STMT_EXECUTE should be classes with close() methods to stop the execution. This will probably mean we need some data purging or something, and it has to be an error to start a new command before a previous has finished. o Rewrite parameter binding code - it is very ugly and hard to understand. The old way was to store Variants and use typeid.toString to find out how to bind the parameter. I've rewritten this to store the parameter in a packet right away so it works with literals, const etc. The code is very ugly though. See packets.ComStmtPrepare o There are stuff I haven't figured out in the prepared statements yet. Cursors and new_params_bound comes to mind. o There might be unused functions in packet_helpers, but I'd like full integration-tests before removing anything. o .. And probably many other stuff
Oct 09 2013